[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:55 dist-test-slave-qb36 krb5kdc[14955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:55 dist-test-slave-qb36 krb5kdc[14955](info): set up 2 sockets
Jun 19 16:48:55 dist-test-slave-qb36 krb5kdc[14955](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 16:48:56.078531 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:48:56 dist-test-slave-qb36 krb5kdc[14955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:48:56.251644 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:44361
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:44361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:56.576481 14968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:56.576799 14968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:56.576972 14968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:56.584229 14968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:56.584340 14968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:56.584432 14968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:56.584538 14968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:56.584621 14968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:56.594807 14968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:44361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:44361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:56.596218 14968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:56.597571 14968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:56.610692 14973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:56.610749 14974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:56.610738 14976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:56.611444 14968 server_base.cc:1034] running on GCE node
I20260619 16:48:56.612244 14968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:56.613731 14968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:56.614959 14968 hybrid_clock.cc:648] HybridClock initialized: now 1781887736614888 us; error 58 us; skew 500 ppm
I20260619 16:48:56.615382 14968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:56 dist-test-slave-qb36 krb5kdc[14955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:56.619136 14968 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:48:56.620697 14968 webserver.cc:460] Webserver started at http://127.14.153.62:44585/ using document root <none> and password file <none>
I20260619 16:48:56.621412 14968 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:56.621516 14968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:56.621776 14968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:56.624899 14968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "a51caf3a8c2d4ed8a194d8d952f87925"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-qb36"
server_key: "2e7e1afc9f7a3cdb915bb6082ce87a85"
server_key_iv: "2daa558dea7ff28d72875e10e6986d30"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.625629 14968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "a51caf3a8c2d4ed8a194d8d952f87925"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-qb36"
server_key: "2e7e1afc9f7a3cdb915bb6082ce87a85"
server_key_iv: "2daa558dea7ff28d72875e10e6986d30"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.631160 14968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:48:56.635069 14983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:56.636461 14968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:56.636650 14968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "a51caf3a8c2d4ed8a194d8d952f87925"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-qb36"
server_key: "2e7e1afc9f7a3cdb915bb6082ce87a85"
server_key_iv: "2daa558dea7ff28d72875e10e6986d30"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.636855 14968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:56.663486 14968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:56.667778 14968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:56.668092 14968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:56.687659 14968 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:44361
I20260619 16:48:56.687664 15034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:44361 every 8 connection(s)
I20260619 16:48:56.689092 14968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:56.692984 15035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:56.699915 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 14968
I20260619 16:48:56.700244 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:48:56.702209 14948 external_mini_cluster.cc:1428] Setting key 045430d6b55016f1bb719c2206c250af
I20260619 16:48:56.703478 15035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925: Bootstrap starting.
I20260619 16:48:56.707446 15035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:56.708662 15035 log.cc:824] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:56.711503 15035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925: No bootstrap required, opened a new log
I20260619 16:48:56.719550 15035 raft_consensus.cc:348] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51caf3a8c2d4ed8a194d8d952f87925" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44361 } }
I20260619 16:48:56.719933 15035 raft_consensus.cc:374] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:56.720077 15035 raft_consensus.cc:729] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51caf3a8c2d4ed8a194d8d952f87925, State: Initialized, Role: FOLLOWER
I20260619 16:48:56.720781 15035 consensus_queue.cc:260] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51caf3a8c2d4ed8a194d8d952f87925" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44361 } }
I20260619 16:48:56.721024 15035 raft_consensus.cc:388] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:56.721181 15035 raft_consensus.cc:482] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:56.721365 15035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:56.723562 15035 raft_consensus.cc:504] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51caf3a8c2d4ed8a194d8d952f87925" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44361 } }
I20260619 16:48:56.724083 15035 leader_election.cc:302] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a51caf3a8c2d4ed8a194d8d952f87925; no voters:
I20260619 16:48:56.724786 15035 leader_election.cc:288] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:56.724934 15040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:56.726728 15035 sys_catalog.cc:564] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:56.730532 15040 raft_consensus.cc:686] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [term 1 LEADER]: Becoming Leader. State: Replica: a51caf3a8c2d4ed8a194d8d952f87925, State: Running, Role: LEADER
I20260619 16:48:56.731168 15040 consensus_queue.cc:237] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51caf3a8c2d4ed8a194d8d952f87925" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44361 } }
I20260619 16:48:56.734651 15041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a51caf3a8c2d4ed8a194d8d952f87925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51caf3a8c2d4ed8a194d8d952f87925" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44361 } } }
I20260619 16:48:56.734674 15042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51caf3a8c2d4ed8a194d8d952f87925. Latest consensus state: current_term: 1 leader_uuid: "a51caf3a8c2d4ed8a194d8d952f87925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51caf3a8c2d4ed8a194d8d952f87925" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44361 } } }
I20260619 16:48:56.735284 15042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:56.735296 15041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925 [sys.catalog]: This master's current role is: LEADER
Jun 19 16:48:56 dist-test-slave-qb36 krb5kdc[14955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:48:56.740226 15050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:56.746230 15050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:56.753919 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:56.706090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.14.153.62:44361)
0619 16:48:56.706902 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:56.706914 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:56.708258 (+ 1344us) server_negotiation.cc:407] Connection header received
0619 16:48:56.718969 (+ 10711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:56.718992 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:56.719482 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:56.719873 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:56.723363 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.725013 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.728346 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.729774 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.730005 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:56.740086 (+ 10081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:56.740103 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:56.740134 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:56.740204 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:56.745528 (+ 5324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:56.746759 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:56.746768 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:56.746776 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:56.747060 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:56.747675 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:56.747682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:56.747692 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:56.748664 (+ 972us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:56.748989 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:56.751496 (+ 2507us) server_negotiation.cc:299] Negotiation successful
0619 16:48:56.752035 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":397,"threads_started":1}
I20260619 16:48:56.759425 15050 catalog_manager.cc:1347] Generated new cluster ID: b47ccd6693424fc8b733f1577507375d
I20260619 16:48:56.759549 15050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:56.772608 15050 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:56.774601 15050 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:56.790525 15050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a51caf3a8c2d4ed8a194d8d952f87925: Generated new TSK 0
I20260619 16:48:56.791972 15050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:48:56.842583 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:44361
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:57.185107 15061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:57.185519 15061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:57.185698 15061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:57.193034 15061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:57.193202 15061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:57.193400 15061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:48:57.204043 15061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:44361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:57.205389 15061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:57.206753 15061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:57.220928 15066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.220865 15067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.220917 15069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:57.221935 15061 server_base.cc:1034] running on GCE node
I20260619 16:48:57.222610 15061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:57.223660 15061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:57.224901 15061 hybrid_clock.cc:648] HybridClock initialized: now 1781887737224855 us; error 40 us; skew 500 ppm
I20260619 16:48:57.225232 15061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:57 dist-test-slave-qb36 krb5kdc[14955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:57.228634 15061 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:48:57.230185 15061 webserver.cc:460] Webserver started at http://127.14.153.1:44331/ using document root <none> and password file <none>
I20260619 16:48:57.230953 15061 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:57.231129 15061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:57.231459 15061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:57.234891 15061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "519e6eeb752a45f3b40b2110ec2a85e7"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-qb36"
server_key: "dd8cf37777aa91d4a0635109cd53f060"
server_key_iv: "857bfd91c76dc30ae1cd3414e9d3b15e"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.235754 15061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "519e6eeb752a45f3b40b2110ec2a85e7"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-qb36"
server_key: "dd8cf37777aa91d4a0635109cd53f060"
server_key_iv: "857bfd91c76dc30ae1cd3414e9d3b15e"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.241068 15061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:48:57.244802 15076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.246295 15061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:48:57.246516 15061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "519e6eeb752a45f3b40b2110ec2a85e7"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-qb36"
server_key: "dd8cf37777aa91d4a0635109cd53f060"
server_key_iv: "857bfd91c76dc30ae1cd3414e9d3b15e"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.246734 15061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:57.270009 15061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:57.274358 15061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:57.274843 15061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:57.276167 15061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:57.277551 15061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:57.277663 15061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.277812 15061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:57.277885 15061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.313506 15061 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:36515
I20260619 16:48:57.313561 15188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:36515 every 8 connection(s)
I20260619 16:48:57.314816 15061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:48:57.322518 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 15061
I20260619 16:48:57.322775 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:48:57.323565 14948 external_mini_cluster.cc:1428] Setting key f7a6d95d5d80bbfe8a497b23e779da4a
Jun 19 16:48:57 dist-test-slave-qb36 krb5kdc[14955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:48:57.337618 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.317438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:32965 (local address 127.14.153.62:44361)
0619 16:48:57.317857 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.317862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.318978 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 16:48:57.321279 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.321285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.321378 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.321516 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:57.323703 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.324577 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.326352 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.326889 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.327017 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.330595 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:57.330604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:57.330612 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:57.330657 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:57.334281 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.335107 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.335115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.335119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.335275 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.335691 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.335696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.335700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.336097 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:57.336265 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.336949 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.337212 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":208,"threads_started":1}
I20260619 16:48:57.338325 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.318046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:44361 (local address 127.14.153.1:32965)
0619 16:48:57.318803 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.318848 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.320974 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.321770 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.321779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.322565 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:57.323478 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.323505 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.324729 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.324735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.326180 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.326192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.327144 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.327152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.327407 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.328215 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:57.328238 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:57.330362 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:57.334436 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.334470 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.334496 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.334941 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.335385 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.335388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.335391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.335565 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.336224 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:57.336230 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:57.336558 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.336789 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.337177 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":307,"threads_started":1}
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:48:57.340050 15189 heartbeater.cc:347] Connected to a master server at 127.14.153.62:44361
I20260619 16:48:57.340387 15189 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:57.341184 15189 heartbeater.cc:511] Master 127.14.153.62:44361 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
I20260619 16:48:57.343560 15000 ts_manager.cc:194] Registered new tserver with Master: 519e6eeb752a45f3b40b2110ec2a85e7 (127.14.153.1:36515)
I20260619 16:48:57.345023 15000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:32965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:48:57.364390 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:44361
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:57.709720 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:57.710002 15195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:57.710173 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:57.717546 15195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:57.717669 15195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:57.717844 15195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:48:57.728778 15195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:44361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:57.730099 15195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:57.731506 15195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:57.745474 15200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.745663 15201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.745474 15203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:57.746958 15195 server_base.cc:1034] running on GCE node
I20260619 16:48:57.747730 15195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:57.748744 15195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:57.749951 15195 hybrid_clock.cc:648] HybridClock initialized: now 1781887737749906 us; error 35 us; skew 500 ppm
I20260619 16:48:57.750322 15195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:57 dist-test-slave-qb36 krb5kdc[14955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:57.753539 15195 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:48:57.755168 15195 webserver.cc:460] Webserver started at http://127.14.153.2:46097/ using document root <none> and password file <none>
I20260619 16:48:57.755822 15195 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:57.755920 15195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:57.756181 15195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:57.759143 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "11fae0ed0acb44b5bef7a502139f750f"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-qb36"
server_key: "57ac6ca0af73e601905c75df8da59c73"
server_key_iv: "bbcf0976f52b842794cbcf19b667073f"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.759871 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "11fae0ed0acb44b5bef7a502139f750f"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-qb36"
server_key: "57ac6ca0af73e601905c75df8da59c73"
server_key_iv: "bbcf0976f52b842794cbcf19b667073f"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.765012 15195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:48:57.768916 15210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.770304 15195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:57.770438 15195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "11fae0ed0acb44b5bef7a502139f750f"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-qb36"
server_key: "57ac6ca0af73e601905c75df8da59c73"
server_key_iv: "bbcf0976f52b842794cbcf19b667073f"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.770603 15195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:57.784343 15195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:57.790192 15195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:57.790506 15195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:57.791551 15195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:57.792848 15195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:57.792917 15195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.793063 15195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:57.793113 15195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.827061 15195 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:38425
I20260619 16:48:57.827080 15322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:38425 every 8 connection(s)
I20260619 16:48:57.828492 15195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:48:57.830852 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 15195
I20260619 16:48:57.831089 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:48:57.831543 14948 external_mini_cluster.cc:1428] Setting key 7d86468a8559cc2bba765ff5a78fb659
Jun 19 16:48:57 dist-test-slave-qb36 krb5kdc[14955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
I20260619 16:48:57.850214 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.831138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:42629 (local address 127.14.153.62:44361)
0619 16:48:57.831333 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.831340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.832724 (+ 1384us) server_negotiation.cc:407] Connection header received
0619 16:48:57.834543 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.834548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.834629 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.834750 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:57.836640 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.837647 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.839281 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.839788 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.839926 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.844238 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:57.844244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:57.844247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:57.844278 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:57.847123 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.847871 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.847877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.847881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.848008 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.848444 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.848449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.848452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.848711 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:57.848867 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.849596 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.849839 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:48:57.851437 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.831771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:44361 (local address 127.14.153.2:42629)
0619 16:48:57.832557 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.832607 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.834275 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.834986 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.834994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.835733 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:57.836459 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.836482 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.837822 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.837825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.839133 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.839149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.840036 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.840043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.840318 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.841059 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:57.841081 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:57.843945 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:57.847265 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.847275 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.847294 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.847727 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.848126 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.848131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.848134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.848324 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.848843 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:57.848851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:57.849190 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.849508 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.850110 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":328,"threads_started":1}
I20260619 16:48:57.852926 15323 heartbeater.cc:347] Connected to a master server at 127.14.153.62:44361
I20260619 16:48:57.853252 15323 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:57.853960 15323 heartbeater.cc:511] Master 127.14.153.62:44361 requested a full tablet report, sending...
I20260619 16:48:57.855433 15000 ts_manager.cc:194] Registered new tserver with Master: 11fae0ed0acb44b5bef7a502139f750f (127.14.153.2:38425)
I20260619 16:48:57.856222 15000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:42629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:48:57.870343 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:44361
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:58.203354 15328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:58.203635 15328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:58.203806 15328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:58.211125 15328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:58.211241 15328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:58.211439 15328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:48:58.221477 15328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:44361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:48:58.222776 15328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:58.224145 15328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:58.237959 15333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:58.238001 15334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:58.238185 15336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:58.239360 15328 server_base.cc:1034] running on GCE node
I20260619 16:48:58.239974 15328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:58.240991 15328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:58.242215 15328 hybrid_clock.cc:648] HybridClock initialized: now 1781887738242154 us; error 57 us; skew 500 ppm
I20260619 16:48:58.242548 15328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[14955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:58.245810 15328 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:48:58.247407 15328 webserver.cc:460] Webserver started at http://127.14.153.3:37435/ using document root <none> and password file <none>
I20260619 16:48:58.248116 15328 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:58.248304 15328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:58.248620 15328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:58.251744 15328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "eb26857dba02424db9f8d5a14b89d8ce"
format_stamp: "Formatted at 2026-06-19 16:48:58 on dist-test-slave-qb36"
server_key: "3a7264d9a4a02f1f1983e91433c46120"
server_key_iv: "81262b08740531d4c5d9a4915612c536"
server_key_version: "encryptionkey@0"
I20260619 16:48:58.252534 15328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "eb26857dba02424db9f8d5a14b89d8ce"
format_stamp: "Formatted at 2026-06-19 16:48:58 on dist-test-slave-qb36"
server_key: "3a7264d9a4a02f1f1983e91433c46120"
server_key_iv: "81262b08740531d4c5d9a4915612c536"
server_key_version: "encryptionkey@0"
I20260619 16:48:58.257764 15328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 16:48:58.261641 15343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.262965 15328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:58.263175 15328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "eb26857dba02424db9f8d5a14b89d8ce"
format_stamp: "Formatted at 2026-06-19 16:48:58 on dist-test-slave-qb36"
server_key: "3a7264d9a4a02f1f1983e91433c46120"
server_key_iv: "81262b08740531d4c5d9a4915612c536"
server_key_version: "encryptionkey@0"
I20260619 16:48:58.263386 15328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:58.285689 15328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:58.290130 15328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:58.290431 15328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:58.291404 15328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:58.292747 15328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:58.292852 15328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.293005 15328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:58.293077 15328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.329964 15328 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:46823
I20260619 16:48:58.330071 15455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:46823 every 8 connection(s)
I20260619 16:48:58.331365 15328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:48:58.338687 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 15328
I20260619 16:48:58.338883 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:48:58.339522 14948 external_mini_cluster.cc:1428] Setting key 10584ef38e8a053533a9c33e19ee4b0a
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[14955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:48:58.348822 15189 heartbeater.cc:503] Master 127.14.153.62:44361 was elected leader, sending a full tablet report...
I20260619 16:48:58.353529 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.334014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:46043 (local address 127.14.153.62:44361)
0619 16:48:58.334214 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.334222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.335488 (+ 1266us) server_negotiation.cc:407] Connection header received
0619 16:48:58.337275 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.337280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.337373 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.337461 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:58.339226 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.340461 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.342796 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.343430 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.343559 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.347306 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:58.347311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:58.347315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:58.347347 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:58.350703 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.351497 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.351501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.351503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.351621 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.352043 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.352046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.352048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.352309 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:58.352413 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.353083 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.353279 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:48:58.354593 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.334560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:44361 (local address 127.14.153.3:46043)
0619 16:48:58.335328 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.335372 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:58.337011 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:58.337668 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:58.337675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:58.338279 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:58.338918 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.338963 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.340706 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.340711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.342628 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.342644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.343695 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.343710 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.343919 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.344675 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:58.344702 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:58.346927 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:58.350842 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:58.350849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:58.350863 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:58.351389 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:58.351713 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:58.351716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:58.351718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:58.351935 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:58.352417 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:58.352423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:58.352750 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 16:48:58.352982 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 16:48:58.353368 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":280,"threads_started":1}
I20260619 16:48:58.356096 15456 heartbeater.cc:347] Connected to a master server at 127.14.153.62:44361
I20260619 16:48:58.356438 15456 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:58.357163 15456 heartbeater.cc:511] Master 127.14.153.62:44361 requested a full tablet report, sending...
I20260619 16:48:58.358536 14999 ts_manager.cc:194] Registered new tserver with Master: eb26857dba02424db9f8d5a14b89d8ce (127.14.153.3:46823)
I20260619 16:48:58.359436 14999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:46043
I20260619 16:48:58.369218 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[14955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[14955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:48:58.412492 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.397714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.14.153.1:36515)
0619 16:48:58.398388 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.398397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.398426 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:48:58.398580 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.398587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.398801 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.398942 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:58.400325 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.401294 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.402483 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.403016 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.403213 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.406952 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:58.406962 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:58.406977 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:58.407008 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:58.409969 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.410572 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.410580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.410582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.410702 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.411103 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.411107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.411109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.411436 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:58.411650 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.411889 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.412071 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":332,"threads_started":1}
W20260619 16:48:58.413816 15104 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33134
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[14955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[14955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:48:58.459308 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.444370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.14.153.1:36515)
0619 16:48:58.444724 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.444759 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.445104 (+ 345us) server_negotiation.cc:407] Connection header received
0619 16:48:58.445368 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.445375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.445495 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.445600 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:58.447457 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.448314 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.449516 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.449966 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.450063 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.454299 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:58.454304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:58.454307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:58.454341 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:58.456797 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.457429 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.457435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.457439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.457560 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.457936 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.457940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.457943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.458208 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:58.458455 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.458748 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.458962 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:48:58.460733 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 15061
I20260619 16:48:58.473634 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.461602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.14.153.1:36515)
0619 16:48:58.461814 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.461820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.462090 (+ 270us) server_negotiation.cc:407] Connection header received
0619 16:48:58.462256 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.462261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.462326 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.462420 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:58.463596 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.464284 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.465520 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.465933 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.466059 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.468820 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:58.468826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:58.468830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:58.468865 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:58.471194 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.471798 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.471804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.471807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.471941 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.472279 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.472286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.472291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.472539 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:58.472784 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.473067 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.473293 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:48:58.474309 15184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:58.557256 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 15061
I20260619 16:48:58.573324 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 15195
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[14955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:48:58.589762 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.574329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.14.153.2:38425)
0619 16:48:58.574946 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.574954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.574979 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:48:58.575094 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.575102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.575332 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.575468 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:58.576600 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.577648 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.578947 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.579609 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.579716 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.583714 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:58.583726 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:58.583740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:58.583772 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:58.586659 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.587623 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.587633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.587635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.587756 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.588182 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.588186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.588188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.588535 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:58.588773 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.589106 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.589400 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":332,"threads_started":1}
I20260619 16:48:58.591013 15318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:58.672858 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 15195
I20260619 16:48:58.689044 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 15328
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[14955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:48:58.706312 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.690117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.14.153.3:46823)
0619 16:48:58.690848 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.690856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.690885 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:48:58.690944 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.690951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.691220 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.691349 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:58.692898 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.693911 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.695885 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.696810 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.696913 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.701152 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:58.701164 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:58.701180 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:58.701221 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:58.703902 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.704531 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.704538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.704540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.704655 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.704950 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.704954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.704956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.705328 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:58.705597 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.705837 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.706029 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":392,"threads_started":1}
I20260619 16:48:58.707788 15451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:58.791097 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 15328
I20260619 16:48:58.806617 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 14968
I20260619 16:48:58.807874 15030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:48:58.908473 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 14968
2026-06-19T16:48:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3931 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[15477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[15477](info): set up 2 sockets
Jun 19 16:48:58 dist-test-slave-qb36 krb5kdc[15477](info): commencing operation
krb5kdc: starting...
W20260619 16:49:00.007977 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:00 dist-test-slave-qb36 krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:49:00.188597 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:45283
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:45283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:00.511212 15490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:00.511557 15490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:00.511718 15490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:00.518994 15490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:00.519140 15490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:00.519187 15490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:00.519340 15490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:00.519405 15490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:00.529423 15490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:45283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:45283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:00.530821 15490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:00.532177 15490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:00.544255 15495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:00.544431 15498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:00.544248 15496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:00.544919 15490 server_base.cc:1034] running on GCE node
I20260619 16:49:00.545845 15490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:00.547261 15490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:00.548461 15490 hybrid_clock.cc:648] HybridClock initialized: now 1781887740548411 us; error 45 us; skew 500 ppm
I20260619 16:49:00.548847 15490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:00 dist-test-slave-qb36 krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887740, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:00.556654 15490 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:49:00.558486 15490 webserver.cc:460] Webserver started at http://127.14.153.62:38073/ using document root <none> and password file <none>
I20260619 16:49:00.559386 15490 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:00.559532 15490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:00.559901 15490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:00.563230 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "852c4ba6ef7945beb0cfa29e2a23d81b"
format_stamp: "Formatted at 2026-06-19 16:49:00 on dist-test-slave-qb36"
server_key: "705d55a6dda48cb2cfcd74514d2329fd"
server_key_iv: "5478a7745f9b0f6a0c6fec62571dc2b7"
server_key_version: "encryptionkey@0"
I20260619 16:49:00.564071 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "852c4ba6ef7945beb0cfa29e2a23d81b"
format_stamp: "Formatted at 2026-06-19 16:49:00 on dist-test-slave-qb36"
server_key: "705d55a6dda48cb2cfcd74514d2329fd"
server_key_iv: "5478a7745f9b0f6a0c6fec62571dc2b7"
server_key_version: "encryptionkey@0"
I20260619 16:49:00.569656 15490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 16:49:00.573573 15505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:00.575160 15490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:00.575485 15490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "852c4ba6ef7945beb0cfa29e2a23d81b"
format_stamp: "Formatted at 2026-06-19 16:49:00 on dist-test-slave-qb36"
server_key: "705d55a6dda48cb2cfcd74514d2329fd"
server_key_iv: "5478a7745f9b0f6a0c6fec62571dc2b7"
server_key_version: "encryptionkey@0"
I20260619 16:49:00.575696 15490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:00.589200 15490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:00.593673 15490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:00.593992 15490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:00.613238 15490 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:45283
I20260619 16:49:00.613267 15556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:45283 every 8 connection(s)
I20260619 16:49:00.614719 15490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:00.615927 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 15490
I20260619 16:49:00.616166 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:00.616811 14948 external_mini_cluster.cc:1428] Setting key 5a777f8cf78ea698e5e75e7b670903d7
I20260619 16:49:00.619561 15557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:49:00 dist-test-slave-qb36 krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:00.633092 15557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b: Bootstrap starting.
I20260619 16:49:00.637320 15557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:00.638585 15557 log.cc:824] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:00.642058 15557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b: No bootstrap required, opened a new log
I20260619 16:49:00.642331 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:00.618995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.14.153.62:45283)
0619 16:49:00.619933 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:00.619950 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:00.620004 (+ 54us) server_negotiation.cc:407] Connection header received
0619 16:49:00.622361 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:00.622391 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:00.622958 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:00.623474 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:00.624837 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.626925 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.628328 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.629368 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.629565 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:00.633868 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:00.633880 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:00.633906 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:00.633949 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:00.637879 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:00.638582 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:00.638591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:00.638601 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:00.638805 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:00.639234 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:00.639242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:00.639248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:00.639835 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:00.640157 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:00.640566 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 16:49:00.640998 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":334,"threads_started":1}
I20260619 16:49:00.650529 15557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852c4ba6ef7945beb0cfa29e2a23d81b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45283 } }
I20260619 16:49:00.650913 15557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:00.650977 15557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852c4ba6ef7945beb0cfa29e2a23d81b, State: Initialized, Role: FOLLOWER
I20260619 16:49:00.651742 15557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852c4ba6ef7945beb0cfa29e2a23d81b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45283 } }
I20260619 16:49:00.652033 15557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:00.652197 15557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:00.652381 15557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:00.654559 15557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852c4ba6ef7945beb0cfa29e2a23d81b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45283 } }
I20260619 16:49:00.655164 15557 leader_election.cc:302] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 852c4ba6ef7945beb0cfa29e2a23d81b; no voters:
I20260619 16:49:00.655794 15557 leader_election.cc:288] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:00.655954 15562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:00.656610 15562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [term 1 LEADER]: Becoming Leader. State: Replica: 852c4ba6ef7945beb0cfa29e2a23d81b, State: Running, Role: LEADER
I20260619 16:49:00.657188 15562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852c4ba6ef7945beb0cfa29e2a23d81b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45283 } }
I20260619 16:49:00.657982 15557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:00.660164 15564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 852c4ba6ef7945beb0cfa29e2a23d81b. Latest consensus state: current_term: 1 leader_uuid: "852c4ba6ef7945beb0cfa29e2a23d81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852c4ba6ef7945beb0cfa29e2a23d81b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45283 } } }
I20260619 16:49:00.660140 15563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "852c4ba6ef7945beb0cfa29e2a23d81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852c4ba6ef7945beb0cfa29e2a23d81b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45283 } } }
I20260619 16:49:00.660601 15563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:00.660601 15564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:00.661608 15568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:00.667312 15568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:00.677210 15568 catalog_manager.cc:1347] Generated new cluster ID: 931f3930de524f31903d4574eebe7c1a
I20260619 16:49:00.677345 15568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:00.702409 15568 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:00.704483 15568 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:00.720542 15568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 852c4ba6ef7945beb0cfa29e2a23d81b: Generated new TSK 0
I20260619 16:49:00.721536 15568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:49:00.778553 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45283
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:01.108292 15583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:01.108592 15583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:01.108758 15583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:01.116130 15583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:01.116250 15583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:01.116454 15583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:01.126679 15583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:01.128420 15583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:01.129781 15583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:01.143527 15589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:01.143733 15591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:01.143507 15588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:01.144557 15583 server_base.cc:1034] running on GCE node
I20260619 16:49:01.145223 15583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:01.146241 15583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:01.147452 15583 hybrid_clock.cc:648] HybridClock initialized: now 1781887741147403 us; error 40 us; skew 500 ppm
I20260619 16:49:01.147807 15583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:01 dist-test-slave-qb36 krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887741, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:01.151213 15583 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:49:01.152793 15583 webserver.cc:460] Webserver started at http://127.14.153.1:35777/ using document root <none> and password file <none>
I20260619 16:49:01.153510 15583 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:01.153640 15583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:01.153971 15583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:01.156960 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "3161e003216141be85cdaf2e8c47a06c"
format_stamp: "Formatted at 2026-06-19 16:49:01 on dist-test-slave-qb36"
server_key: "02d36934701dc7df033db8454c216d91"
server_key_iv: "fa6b6d2c8fbcd76a1e726fbd18dd2783"
server_key_version: "encryptionkey@0"
I20260619 16:49:01.157785 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "3161e003216141be85cdaf2e8c47a06c"
format_stamp: "Formatted at 2026-06-19 16:49:01 on dist-test-slave-qb36"
server_key: "02d36934701dc7df033db8454c216d91"
server_key_iv: "fa6b6d2c8fbcd76a1e726fbd18dd2783"
server_key_version: "encryptionkey@0"
I20260619 16:49:01.163296 15583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 16:49:01.166834 15598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:01.168303 15583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:01.168485 15583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "3161e003216141be85cdaf2e8c47a06c"
format_stamp: "Formatted at 2026-06-19 16:49:01 on dist-test-slave-qb36"
server_key: "02d36934701dc7df033db8454c216d91"
server_key_iv: "fa6b6d2c8fbcd76a1e726fbd18dd2783"
server_key_version: "encryptionkey@0"
I20260619 16:49:01.168701 15583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:01.192651 15583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:01.196735 15583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:01.197033 15583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:01.198045 15583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:01.199409 15583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:01.199481 15583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:01.199622 15583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:01.199705 15583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:01.232722 15583 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:42307
I20260619 16:49:01.232795 15710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:42307 every 8 connection(s)
I20260619 16:49:01.234030 15583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:01.236208 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 15583
I20260619 16:49:01.236423 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:01.237167 14948 external_mini_cluster.cc:1428] Setting key 28f9431e5a37edf52917926f660b47bb
Jun 19 16:49:01 dist-test-slave-qb36 krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887741, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
I20260619 16:49:01.255092 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:01.236813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:55001 (local address 127.14.153.62:45283)
0619 16:49:01.237349 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:01.237356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:01.238463 (+ 1107us) server_negotiation.cc:407] Connection header received
0619 16:49:01.240343 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:01.240351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:01.240432 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:01.240559 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:01.242054 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:01.243013 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:01.244623 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:01.245022 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:01.245164 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:01.248945 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:01.248950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:01.248954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:01.248985 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:01.251658 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:01.252530 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:01.252536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:01.252541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:01.252662 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:01.253124 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:01.253129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:01.253133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:01.253412 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:01.253562 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:01.254400 (+ 838us) server_negotiation.cc:299] Negotiation successful
0619 16:49:01.254685 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":252,"threads_started":1}
I20260619 16:49:01.256129 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:01.237472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45283 (local address 127.14.153.1:55001)
0619 16:49:01.238287 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:01.238334 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:01.240066 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:01.240795 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:01.240803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:01.241317 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:01.241887 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:01.241904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:01.243209 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:01.243220 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:01.244467 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:01.244480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:01.245255 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:01.245264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:01.245505 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:01.246212 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:01.246235 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:01.248704 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:01.251826 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:01.251837 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:01.251858 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:01.252401 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:01.252786 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:01.252793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:01.252796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:01.252994 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:01.253539 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:01.253548 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:01.253926 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 16:49:01.254300 (+ 374us) client_negotiation.cc:240] Negotiation successful
0619 16:49:01.254864 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":334,"threads_started":1}
I20260619 16:49:01.257889 15711 heartbeater.cc:347] Connected to a master server at 127.14.153.62:45283
I20260619 16:49:01.258260 15711 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:01.259140 15711 heartbeater.cc:511] Master 127.14.153.62:45283 requested a full tablet report, sending...
I20260619 16:49:01.261346 15522 ts_manager.cc:194] Registered new tserver with Master: 3161e003216141be85cdaf2e8c47a06c (127.14.153.1:42307)
I20260619 16:49:01.262792 15522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:55001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:49:01.277878 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45283
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:01.600194 15717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:01.600538 15717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:01.600715 15717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:01.607954 15717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:01.608143 15717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:01.608333 15717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:01.622257 15717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:01.623931 15717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:01.625519 15717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:01.640486 15722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:01.640532 15723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:01.640460 15725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:01.641274 15717 server_base.cc:1034] running on GCE node
I20260619 16:49:01.641844 15717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:01.642786 15717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:01.643988 15717 hybrid_clock.cc:648] HybridClock initialized: now 1781887741643941 us; error 35 us; skew 500 ppm
I20260619 16:49:01.644268 15717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:01 dist-test-slave-qb36 krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887741, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:01.647244 15717 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:49:01.648686 15717 webserver.cc:460] Webserver started at http://127.14.153.2:42149/ using document root <none> and password file <none>
I20260619 16:49:01.649407 15717 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:01.649547 15717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:01.649873 15717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:01.652932 15717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "7de917c63edf45bca78c3c8c407610f8"
format_stamp: "Formatted at 2026-06-19 16:49:01 on dist-test-slave-qb36"
server_key: "0248e313e570e49ef647ef27b239947a"
server_key_iv: "1e21d3c121f52411a86599b37af06c86"
server_key_version: "encryptionkey@0"
I20260619 16:49:01.653843 15717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "7de917c63edf45bca78c3c8c407610f8"
format_stamp: "Formatted at 2026-06-19 16:49:01 on dist-test-slave-qb36"
server_key: "0248e313e570e49ef647ef27b239947a"
server_key_iv: "1e21d3c121f52411a86599b37af06c86"
server_key_version: "encryptionkey@0"
I20260619 16:49:01.659319 15717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:49:01.662705 15732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:01.664252 15717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:01.664425 15717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "7de917c63edf45bca78c3c8c407610f8"
format_stamp: "Formatted at 2026-06-19 16:49:01 on dist-test-slave-qb36"
server_key: "0248e313e570e49ef647ef27b239947a"
server_key_iv: "1e21d3c121f52411a86599b37af06c86"
server_key_version: "encryptionkey@0"
I20260619 16:49:01.664618 15717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:01.689745 15717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:01.694336 15717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:01.694623 15717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:01.695681 15717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:01.697029 15717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:01.697134 15717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:01.697285 15717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:01.697356 15717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:01.729475 15717 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:39583
I20260619 16:49:01.729563 15844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:39583 every 8 connection(s)
I20260619 16:49:01.730762 15717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:01.735286 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 15717
I20260619 16:49:01.735481 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:01.736042 14948 external_mini_cluster.cc:1428] Setting key 2862c939cf5aceb4dc6dc50d9813be50
Jun 19 16:49:01 dist-test-slave-qb36 krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887741, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:01.753660 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:01.733404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:56485 (local address 127.14.153.62:45283)
0619 16:49:01.733588 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:01.733593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:01.734753 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 16:49:01.737347 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:01.737352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:01.737433 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:01.737550 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:01.739459 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:01.740129 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:01.742629 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:01.743097 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:01.743195 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:01.746714 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:01.746723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:01.746731 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:01.746772 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:01.750212 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:01.751234 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:01.751239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:01.751241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:01.751381 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:01.751887 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:01.751890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:01.751892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:01.752142 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:01.752262 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:01.753151 (+ 889us) server_negotiation.cc:299] Negotiation successful
0619 16:49:01.753336 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:01.754993 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:01.733880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45283 (local address 127.14.153.2:56485)
0619 16:49:01.734564 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:01.734608 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:01.737057 (+ 2449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:01.737791 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:01.737799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:01.738415 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:01.739271 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:01.739298 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:01.740282 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:01.740291 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:01.742483 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:01.742501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:01.743317 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:01.743325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:01.743650 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:01.744429 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:01.744449 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:01.746482 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:01.750353 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:01.750365 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:01.750386 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:01.751116 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:01.751495 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:01.751501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:01.751506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:01.751777 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:01.752280 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:01.752290 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:01.752757 (+ 467us) client_negotiation.cc:769] Sending connection context
0619 16:49:01.753082 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 16:49:01.753684 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":289,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
I20260619 16:49:01.756543 15845 heartbeater.cc:347] Connected to a master server at 127.14.153.62:45283
I20260619 16:49:01.756888 15845 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:01.757669 15845 heartbeater.cc:511] Master 127.14.153.62:45283 requested a full tablet report, sending...
I20260619 16:49:01.759382 15522 ts_manager.cc:194] Registered new tserver with Master: 7de917c63edf45bca78c3c8c407610f8 (127.14.153.2:39583)
I20260619 16:49:01.760263 15522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:56485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:49:01.779284 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45283
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:02.107147 15850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:02.107427 15850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:02.107606 15850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:02.114825 15850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:02.114933 15850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:02.115150 15850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:02.124953 15850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:02.126281 15850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:02.127784 15850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:02.141041 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:02.141052 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:02.141165 15855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:02.141845 15850 server_base.cc:1034] running on GCE node
I20260619 16:49:02.142503 15850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:02.143524 15850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:02.144758 15850 hybrid_clock.cc:648] HybridClock initialized: now 1781887742144719 us; error 55 us; skew 500 ppm
I20260619 16:49:02.145088 15850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:02 dist-test-slave-qb36 krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887742, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:02.148288 15850 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:49:02.149797 15850 webserver.cc:460] Webserver started at http://127.14.153.3:36297/ using document root <none> and password file <none>
I20260619 16:49:02.150508 15850 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:02.150651 15850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:02.151093 15850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:02.154068 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "6d484233bb6e4e77ab0cc65c60bb4565"
format_stamp: "Formatted at 2026-06-19 16:49:02 on dist-test-slave-qb36"
server_key: "f83527943f3fe7283902718e42a7257f"
server_key_iv: "a01450f34f00829930fa0da8747c9e50"
server_key_version: "encryptionkey@0"
I20260619 16:49:02.155128 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "6d484233bb6e4e77ab0cc65c60bb4565"
format_stamp: "Formatted at 2026-06-19 16:49:02 on dist-test-slave-qb36"
server_key: "f83527943f3fe7283902718e42a7257f"
server_key_iv: "a01450f34f00829930fa0da8747c9e50"
server_key_version: "encryptionkey@0"
I20260619 16:49:02.160477 15850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:49:02.164116 15865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:02.165459 15850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:49:02.165633 15850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "6d484233bb6e4e77ab0cc65c60bb4565"
format_stamp: "Formatted at 2026-06-19 16:49:02 on dist-test-slave-qb36"
server_key: "f83527943f3fe7283902718e42a7257f"
server_key_iv: "a01450f34f00829930fa0da8747c9e50"
server_key_version: "encryptionkey@0"
I20260619 16:49:02.165850 15850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:02.191994 15850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:02.196626 15850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:02.196935 15850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:02.198014 15850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:02.199422 15850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:02.199524 15850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:02.199673 15850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:02.199745 15850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:02.232976 15850 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:40393
I20260619 16:49:02.233008 15977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:40393 every 8 connection(s)
I20260619 16:49:02.234275 15850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:02.239050 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 15850
I20260619 16:49:02.239246 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:02.239794 14948 external_mini_cluster.cc:1428] Setting key d21f0dbe1515cd0213285ba4688d0f55
Jun 19 16:49:02 dist-test-slave-qb36 krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887742, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:02.255300 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.236983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:51273 (local address 127.14.153.62:45283)
0619 16:49:02.237191 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.237197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.238424 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 16:49:02.240949 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.240954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.241048 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.241179 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:02.243118 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.243846 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.245306 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.245670 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.245759 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.248994 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:02.248998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:02.249002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:02.249032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:02.251653 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.252448 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.252452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.252454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.252574 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.253070 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.253074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.253076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.253513 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:02.253642 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.254624 (+ 982us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.254845 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:49:02.256295 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.237557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45283 (local address 127.14.153.3:51273)
0619 16:49:02.238253 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:02.238293 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:02.240638 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:02.241475 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:02.241488 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:02.242155 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:02.242924 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.242945 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.243975 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.243978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.245190 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.245201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.245854 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.245858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.246039 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.246642 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:02.246662 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:02.248820 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:02.251805 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:02.251812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:02.251825 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:02.252314 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:02.252702 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:02.252706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:02.252708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:02.252894 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:02.253659 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:02.253668 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:02.254118 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 16:49:02.254467 (+ 349us) client_negotiation.cc:240] Negotiation successful
0619 16:49:02.254978 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":278,"threads_started":1}
I20260619 16:49:02.258018 15978 heartbeater.cc:347] Connected to a master server at 127.14.153.62:45283
I20260619 16:49:02.258347 15978 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:02.259085 15978 heartbeater.cc:511] Master 127.14.153.62:45283 requested a full tablet report, sending...
I20260619 16:49:02.260496 15522 ts_manager.cc:194] Registered new tserver with Master: 6d484233bb6e4e77ab0cc65c60bb4565 (127.14.153.3:40393)
I20260619 16:49:02.261243 15522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:51273
I20260619 16:49:02.266712 15711 heartbeater.cc:503] Master 127.14.153.62:45283 was elected leader, sending a full tablet report...
I20260619 16:49:02.268136 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:02.287631 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.274831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.14.153.62:45283)
0619 16:49:02.275070 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.275079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.275540 (+ 461us) server_negotiation.cc:407] Connection header received
0619 16:49:02.275659 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.275662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.275720 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.275803 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:02.277329 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.278095 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.279388 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.279848 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.279960 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.282704 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:02.282708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:02.282711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:02.282753 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:02.285175 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.285797 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.285800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.285803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.285923 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.286249 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.286252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.286254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.286547 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:02.286716 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.286979 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.287357 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:49:02.294061 15521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:02.300794 15521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:02.330354 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.312712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:42307 (local address 127.0.0.1:52604)
0619 16:49:02.313654 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:02.313700 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:02.313885 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:02.316625 (+ 2740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:02.316637 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:02.316682 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:02.317246 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.317260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.321315 (+ 4055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.321325 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.324668 (+ 3343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.324690 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.328901 (+ 4211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.328911 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.329218 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.329305 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 16:49:02.329447 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:49:02.329717 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":26,"thread_start_us":549,"threads_started":1}
I20260619 16:49:02.332155 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.312909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:40393 (local address 127.0.0.1:54198)
0619 16:49:02.317033 (+ 4124us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:02.317047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:02.317201 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:02.318310 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:02.318317 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:02.318352 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:02.318882 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.318896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.321558 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.321565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.327675 (+ 6110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.327695 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.330946 (+ 3251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.330956 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.331246 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.331301 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 16:49:02.331412 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:49:02.331602 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4026,"mutex_wait_us":4,"thread_start_us":3892,"threads_started":1}
I20260619 16:49:02.333070 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.313504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.14.153.3:40393)
0619 16:49:02.315601 (+ 2097us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.315614 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.317153 (+ 1539us) server_negotiation.cc:407] Connection header received
0619 16:49:02.317440 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.317448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.317702 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.317922 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:02.319138 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.321279 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.327926 (+ 6647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.330402 (+ 2476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.330539 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.330762 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.332566 (+ 1804us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.332752 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1899,"thread_start_us":1746,"threads_started":1}
I20260619 16:49:02.333814 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.313033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.14.153.1:42307)
0619 16:49:02.316083 (+ 3050us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.316089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.316113 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:49:02.316195 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.316201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.316395 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.316554 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:02.317461 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.321103 (+ 3642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.324902 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.328585 (+ 3683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.331415 (+ 2830us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.333071 (+ 1656us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.333180 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.333414 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2911,"thread_start_us":2789,"threads_started":1}
I20260619 16:49:02.337877 15913 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4189dcbe724885bb7810cb87dba804 (DEFAULT_TABLE table=test-table [id=5833aedd7a134718a20e093de41b3ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:02.337891 15646 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4189dcbe724885bb7810cb87dba804 (DEFAULT_TABLE table=test-table [id=5833aedd7a134718a20e093de41b3ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:02.339211 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.317399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:39583 (local address 127.0.0.1:44912)
0619 16:49:02.331018 (+ 13619us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:02.331063 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:02.331197 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:02.333826 (+ 2629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:02.333834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:02.333864 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:02.334065 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.334082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.335915 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.335921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.337040 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.337050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.338458 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.338464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.338651 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.338683 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:49:02.338765 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:49:02.338904 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13471,"thread_start_us":16266,"threads_started":1}
I20260619 16:49:02.339620 15646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4189dcbe724885bb7810cb87dba804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:02.339902 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.317446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.14.153.2:39583)
0619 16:49:02.332906 (+ 15460us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.332915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.332948 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:49:02.333054 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.333066 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.333359 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.334216 (+ 857us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:02.334391 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.335775 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.337195 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.338281 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.339106 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.339351 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.339425 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.339600 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15289,"thread_start_us":16083,"threads_started":1}
I20260619 16:49:02.339970 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4189dcbe724885bb7810cb87dba804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:02.342958 15780 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4189dcbe724885bb7810cb87dba804 (DEFAULT_TABLE table=test-table [id=5833aedd7a134718a20e093de41b3ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:02.344143 15780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4189dcbe724885bb7810cb87dba804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:02.356326 15998 tablet_bootstrap.cc:492] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565: Bootstrap starting.
I20260619 16:49:02.359745 15997 tablet_bootstrap.cc:492] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c: Bootstrap starting.
I20260619 16:49:02.358959 15996 tablet_bootstrap.cc:492] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8: Bootstrap starting.
I20260619 16:49:02.360646 15998 tablet_bootstrap.cc:654] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:02.362066 15998 log.cc:824] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:02.363566 15996 tablet_bootstrap.cc:654] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:02.363564 15997 tablet_bootstrap.cc:654] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:02.364866 15997 log.cc:824] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:02.364866 15996 log.cc:824] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:02.371450 15996 tablet_bootstrap.cc:492] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8: No bootstrap required, opened a new log
I20260619 16:49:02.371671 15998 tablet_bootstrap.cc:492] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565: No bootstrap required, opened a new log
I20260619 16:49:02.371763 15996 ts_tablet_manager.cc:1397] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260619 16:49:02.371958 15998 ts_tablet_manager.cc:1397] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20260619 16:49:02.378896 15996 raft_consensus.cc:348] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } }
I20260619 16:49:02.379788 15996 raft_consensus.cc:374] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:02.379966 15996 raft_consensus.cc:729] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de917c63edf45bca78c3c8c407610f8, State: Initialized, Role: FOLLOWER
I20260619 16:49:02.380034 15998 raft_consensus.cc:348] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } }
I20260619 16:49:02.380477 15998 raft_consensus.cc:374] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:02.380586 15998 raft_consensus.cc:729] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d484233bb6e4e77ab0cc65c60bb4565, State: Initialized, Role: FOLLOWER
I20260619 16:49:02.380851 15996 consensus_queue.cc:260] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } }
I20260619 16:49:02.381173 15997 tablet_bootstrap.cc:492] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c: No bootstrap required, opened a new log
I20260619 16:49:02.381268 15998 consensus_queue.cc:260] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } }
I20260619 16:49:02.381471 15997 ts_tablet_manager.cc:1397] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.002s
I20260619 16:49:02.382030 15845 heartbeater.cc:503] Master 127.14.153.62:45283 was elected leader, sending a full tablet report...
I20260619 16:49:02.383839 15996 ts_tablet_manager.cc:1428] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260619 16:49:02.384703 15978 heartbeater.cc:503] Master 127.14.153.62:45283 was elected leader, sending a full tablet report...
I20260619 16:49:02.386978 15998 ts_tablet_manager.cc:1428] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565: Time spent starting tablet: real 0.015s user 0.004s sys 0.010s
I20260619 16:49:02.391820 15997 raft_consensus.cc:348] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } }
I20260619 16:49:02.392325 15997 raft_consensus.cc:374] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:02.392457 15997 raft_consensus.cc:729] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3161e003216141be85cdaf2e8c47a06c, State: Initialized, Role: FOLLOWER
I20260619 16:49:02.393198 15997 consensus_queue.cc:260] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } }
I20260619 16:49:02.395743 15997 ts_tablet_manager.cc:1428] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260619 16:49:02.441064 16002 raft_consensus.cc:482] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:02.441349 16002 raft_consensus.cc:504] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } }
I20260619 16:49:02.442979 16002 leader_election.cc:288] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3161e003216141be85cdaf2e8c47a06c (127.14.153.1:42307), 6d484233bb6e4e77ab0cc65c60bb4565 (127.14.153.3:40393)
I20260619 16:49:02.449712 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.443292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:32999 (local address 127.14.153.1:42307)
0619 16:49:02.443569 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.443578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.444283 (+ 705us) server_negotiation.cc:407] Connection header received
0619 16:49:02.444467 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.444475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.444559 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.444625 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:02.445283 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.446020 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.447978 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.448785 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.448974 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.449048 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.449322 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.449443 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:49:02.449877 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.443208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:42307 (local address 127.14.153.2:32999)
0619 16:49:02.444150 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:02.444165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:02.444293 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:02.444672 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:02.444679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:02.444716 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:02.445128 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.445138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.446152 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.446157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.447803 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.447814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.448994 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.449000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.449159 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.449182 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:49:02.449300 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 16:49:02.449424 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":23,"thread_start_us":555,"threads_started":1}
I20260619 16:49:02.450294 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.443530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:54803 (local address 127.14.153.3:40393)
0619 16:49:02.443765 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.443772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.444378 (+ 606us) server_negotiation.cc:407] Connection header received
0619 16:49:02.444544 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.444548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.444631 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.444720 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:02.445443 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.446195 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.447946 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.449333 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.449512 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.449589 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.449961 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.450106 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:49:02.450829 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.443259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:40393 (local address 127.14.153.2:54803)
0619 16:49:02.444201 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:02.444252 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:02.444419 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:02.444757 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:02.444761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:02.444855 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:02.445287 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.445299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.446330 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.446334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.447794 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:02.447806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.449577 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.449584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:02.449755 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.449776 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:49:02.450466 (+ 690us) client_negotiation.cc:240] Negotiation successful
0619 16:49:02.450570 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":50,"spinlock_wait_cycles":25344,"thread_start_us":676,"threads_started":1}
I20260619 16:49:02.450806 15666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4189dcbe724885bb7810cb87dba804" candidate_uuid: "7de917c63edf45bca78c3c8c407610f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3161e003216141be85cdaf2e8c47a06c" is_pre_election: true
I20260619 16:49:02.451498 15666 raft_consensus.cc:2454] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7de917c63edf45bca78c3c8c407610f8 in term 0.
I20260619 16:49:02.451834 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4189dcbe724885bb7810cb87dba804" candidate_uuid: "7de917c63edf45bca78c3c8c407610f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" is_pre_election: true
I20260619 16:49:02.452230 15736 leader_election.cc:302] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3161e003216141be85cdaf2e8c47a06c, 7de917c63edf45bca78c3c8c407610f8; no voters:
I20260619 16:49:02.452361 15933 raft_consensus.cc:2454] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7de917c63edf45bca78c3c8c407610f8 in term 0.
I20260619 16:49:02.452744 16002 raft_consensus.cc:2781] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:02.452912 16002 raft_consensus.cc:482] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:02.453009 16002 raft_consensus.cc:3037] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:02.455215 16002 raft_consensus.cc:504] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } }
I20260619 16:49:02.455848 16002 leader_election.cc:288] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [CANDIDATE]: Term 1 election: Requested vote from peers 3161e003216141be85cdaf2e8c47a06c (127.14.153.1:42307), 6d484233bb6e4e77ab0cc65c60bb4565 (127.14.153.3:40393)
I20260619 16:49:02.456244 15666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4189dcbe724885bb7810cb87dba804" candidate_uuid: "7de917c63edf45bca78c3c8c407610f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3161e003216141be85cdaf2e8c47a06c"
I20260619 16:49:02.456246 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4189dcbe724885bb7810cb87dba804" candidate_uuid: "7de917c63edf45bca78c3c8c407610f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d484233bb6e4e77ab0cc65c60bb4565"
I20260619 16:49:02.456449 15933 raft_consensus.cc:3037] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:02.456444 15666 raft_consensus.cc:3037] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:02.458819 15666 raft_consensus.cc:2454] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de917c63edf45bca78c3c8c407610f8 in term 1.
I20260619 16:49:02.459388 15933 raft_consensus.cc:2454] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de917c63edf45bca78c3c8c407610f8 in term 1.
I20260619 16:49:02.459350 15736 leader_election.cc:302] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3161e003216141be85cdaf2e8c47a06c, 7de917c63edf45bca78c3c8c407610f8; no voters:
I20260619 16:49:02.460094 16002 raft_consensus.cc:2781] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:02.460911 16002 raft_consensus.cc:686] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [term 1 LEADER]: Becoming Leader. State: Replica: 7de917c63edf45bca78c3c8c407610f8, State: Running, Role: LEADER
I20260619 16:49:02.461463 16002 consensus_queue.cc:237] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } }
I20260619 16:49:02.467015 15522 catalog_manager.cc:5495] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7de917c63edf45bca78c3c8c407610f8 (127.14.153.2). New cstate: current_term: 1 leader_uuid: "7de917c63edf45bca78c3c8c407610f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3161e003216141be85cdaf2e8c47a06c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d484233bb6e4e77ab0cc65c60bb4565" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7de917c63edf45bca78c3c8c407610f8" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39583 } health_report { overall_health: HEALTHY } } }
Jun 19 16:49:02 dist-test-slave-qb36 krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
W20260619 16:49:02.482781 15846 tablet.cc:2369] T 0b4189dcbe724885bb7810cb87dba804 P 7de917c63edf45bca78c3c8c407610f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:02.486030 15979 tablet.cc:2369] T 0b4189dcbe724885bb7810cb87dba804 P 6d484233bb6e4e77ab0cc65c60bb4565: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:02.486400 15712 tablet.cc:2369] T 0b4189dcbe724885bb7810cb87dba804 P 3161e003216141be85cdaf2e8c47a06c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:02.488461 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.473004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.14.153.1:42307)
0619 16:49:02.473159 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.473164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.473750 (+ 586us) server_negotiation.cc:407] Connection header received
0619 16:49:02.473960 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.473964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.474035 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.474140 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:02.475682 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.476423 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.477637 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.478055 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.478207 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.482220 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:02.482233 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:02.482248 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:02.482289 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:02.485412 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.486050 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.486062 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.486067 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.486236 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.486727 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.486731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.486734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.487579 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:02.487806 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.488060 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.488218 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 16:49:02 dist-test-slave-qb36 krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:49:02 dist-test-slave-qb36 krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:02.531383 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.518286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.14.153.1:42307)
0619 16:49:02.518517 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.518525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.518787 (+ 262us) server_negotiation.cc:407] Connection header received
0619 16:49:02.519012 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.519016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.519141 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.519245 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:02.520547 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.521272 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.522530 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.522949 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.523118 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.526777 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:02.526782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:02.526785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:02.526818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:02.529257 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.529811 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.529814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.529816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.529932 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.530278 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.530281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.530283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.530520 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:02.530694 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.530914 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.531139 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 16:49:02 dist-test-slave-qb36 krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:49:02 dist-test-slave-qb36 krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:02.575868 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.561673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.14.153.1:42307)
0619 16:49:02.561886 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.561896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.562208 (+ 312us) server_negotiation.cc:407] Connection header received
0619 16:49:02.562460 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.562464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.562528 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.562636 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:02.564026 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.564691 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.565888 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.566313 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.566444 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.570888 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:02.570893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:02.570896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:02.570937 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:02.573589 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.574253 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.574257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.574260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.574373 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.574663 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.574666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.574668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.574966 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:02.575222 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.575435 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.575602 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 16:49:02.577641 15626 tablet_service.cc:2908] Rejecting scan request for tablet 0b4189dcbe724885bb7810cb87dba804: Uninitialized: clean time has not yet been initialized
I20260619 16:49:02.580862 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 15583
I20260619 16:49:02.592895 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.581621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.14.153.1:42307)
0619 16:49:02.581774 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.581778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.582115 (+ 337us) server_negotiation.cc:407] Connection header received
0619 16:49:02.582223 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.582226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.582295 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.582373 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:02.583544 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.584201 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.585326 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.585706 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.585816 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.588028 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:02.588032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:02.588035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:02.588062 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:02.590737 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.591429 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.591432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.591435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.591545 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.591842 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.591845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.591847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.592103 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:02.592303 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.592501 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.592664 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:49:02.593582 15706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:02.682552 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 15583
I20260619 16:49:02.699584 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 15717
Jun 19 16:49:02 dist-test-slave-qb36 krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:49:02.713699 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.700324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.14.153.2:39583)
0619 16:49:02.700502 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.700506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.700523 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:49:02.700617 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.700621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.700679 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.700771 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:02.701837 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.702507 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.703826 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.704213 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.704309 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.708326 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:02.708342 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:02.708360 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:02.708394 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:02.710997 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.711697 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.711713 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.711716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.711842 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.712233 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.712242 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.712244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.712563 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:02.712868 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.713135 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.713360 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:49:02.714324 15840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:02.802996 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 15717
I20260619 16:49:02.819564 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 15850
Jun 19 16:49:02 dist-test-slave-qb36 krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:49:02.835005 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.820376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.14.153.3:40393)
0619 16:49:02.820554 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:02.820562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:02.820625 (+ 63us) server_negotiation.cc:407] Connection header received
0619 16:49:02.820822 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:02.820828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:02.820914 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:02.821025 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:02.822239 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.822979 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.824321 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:02.824744 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:02.824875 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:02.829435 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:02.829452 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:02.829469 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:02.829508 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:02.832178 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.832888 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.832896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.832898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.833033 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:02.833427 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:02.833433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:02.833436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:02.833834 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:02.834135 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:02.834463 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 16:49:02.834681 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:49:02.835742 15973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:02.926275 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 15850
I20260619 16:49:02.942049 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 15490
I20260619 16:49:02.943346 15552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:03.048623 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 15490
2026-06-19T16:49:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4141 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:03 dist-test-slave-qb36 krb5kdc[16026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:03 dist-test-slave-qb36 krb5kdc[16026](info): set up 2 sockets
Jun 19 16:49:03 dist-test-slave-qb36 krb5kdc[16026](info): commencing operation
krb5kdc: starting...
W20260619 16:49:04.135880 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:04 dist-test-slave-qb36 krb5kdc[16026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:49:04.316263 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:39261
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:04.633911 16039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:04.634217 16039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:04.634393 16039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:04.641616 16039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:04.641716 16039 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:49:04.641819 16039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:04.641929 16039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:04.642027 16039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:04.642132 16039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:04.652032 16039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:39261
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:04.653358 16039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:04.654736 16039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:04.667194 16047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:04.667220 16045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:04.667613 16044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:04.668138 16039 server_base.cc:1034] running on GCE node
I20260619 16:49:04.668987 16039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:04.670512 16039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:04.671744 16039 hybrid_clock.cc:648] HybridClock initialized: now 1781887744671659 us; error 59 us; skew 500 ppm
I20260619 16:49:04.672147 16039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:04 dist-test-slave-qb36 krb5kdc[16026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887744, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:04.675273 16039 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:49:04.676723 16039 webserver.cc:460] Webserver started at http://127.14.153.62:33173/ using document root <none> and password file <none>
I20260619 16:49:04.677537 16039 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:04.677677 16039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:04.678078 16039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:04.681234 16039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "384070a0e8b54d7f8ca0e21667a486cc"
format_stamp: "Formatted at 2026-06-19 16:49:04 on dist-test-slave-qb36"
server_key: "f841a1979de2733e487751154b340227"
server_key_iv: "42a3908fa1ed2a28ce9b2d17ab902bb7"
server_key_version: "encryptionkey@0"
I20260619 16:49:04.681963 16039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "384070a0e8b54d7f8ca0e21667a486cc"
format_stamp: "Formatted at 2026-06-19 16:49:04 on dist-test-slave-qb36"
server_key: "f841a1979de2733e487751154b340227"
server_key_iv: "42a3908fa1ed2a28ce9b2d17ab902bb7"
server_key_version: "encryptionkey@0"
I20260619 16:49:04.687273 16039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:49:04.690706 16054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:04.692147 16039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:04.692373 16039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "384070a0e8b54d7f8ca0e21667a486cc"
format_stamp: "Formatted at 2026-06-19 16:49:04 on dist-test-slave-qb36"
server_key: "f841a1979de2733e487751154b340227"
server_key_iv: "42a3908fa1ed2a28ce9b2d17ab902bb7"
server_key_version: "encryptionkey@0"
I20260619 16:49:04.692600 16039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:04.722169 16039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:04.726315 16039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:04.726611 16039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:04.747300 16039 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:39261
I20260619 16:49:04.747321 16115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:39261 every 8 connection(s)
I20260619 16:49:04.748661 16039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:04.752301 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:04.753618 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 16039
I20260619 16:49:04.753882 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:04.754550 14948 external_mini_cluster.cc:1428] Setting key d26b8bbdb7c85914625d7b3f611e280d
I20260619 16:49:04.766515 16116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc: Bootstrap starting.
Jun 19 16:49:04 dist-test-slave-qb36 krb5kdc[16026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:04.770437 16116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:04.771867 16116 log.cc:824] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:04.775256 16116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc: No bootstrap required, opened a new log
I20260619 16:49:04.778098 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:04.756594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.14.153.62:39261)
0619 16:49:04.757415 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:04.757427 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:04.757470 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:49:04.759179 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:04.759209 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:04.759633 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:04.759975 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:04.761739 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:04.763189 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:04.764950 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:04.765611 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:04.765765 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:04.770351 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:04.770364 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:04.770384 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:04.770443 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:04.774161 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:04.774811 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:04.774819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:04.774828 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:04.775018 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:04.775396 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:04.775404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:04.775409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:04.775924 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:04.776267 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:04.776648 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 16:49:04.777081 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":380,"threads_started":1}
I20260619 16:49:04.783955 16116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384070a0e8b54d7f8ca0e21667a486cc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 39261 } }
I20260619 16:49:04.784432 16116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:04.784551 16116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384070a0e8b54d7f8ca0e21667a486cc, State: Initialized, Role: FOLLOWER
I20260619 16:49:04.785195 16116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384070a0e8b54d7f8ca0e21667a486cc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 39261 } }
I20260619 16:49:04.785439 16116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:04.785574 16116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:04.785746 16116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:04.788246 16116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384070a0e8b54d7f8ca0e21667a486cc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 39261 } }
I20260619 16:49:04.788781 16116 leader_election.cc:302] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384070a0e8b54d7f8ca0e21667a486cc; no voters:
I20260619 16:49:04.789418 16116 leader_election.cc:288] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:04.789577 16121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:04.790346 16121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [term 1 LEADER]: Becoming Leader. State: Replica: 384070a0e8b54d7f8ca0e21667a486cc, State: Running, Role: LEADER
I20260619 16:49:04.790921 16121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384070a0e8b54d7f8ca0e21667a486cc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 39261 } }
I20260619 16:49:04.791694 16116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:04.793613 16123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384070a0e8b54d7f8ca0e21667a486cc. Latest consensus state: current_term: 1 leader_uuid: "384070a0e8b54d7f8ca0e21667a486cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384070a0e8b54d7f8ca0e21667a486cc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 39261 } } }
I20260619 16:49:04.793962 16123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:04.794565 16122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384070a0e8b54d7f8ca0e21667a486cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384070a0e8b54d7f8ca0e21667a486cc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 39261 } } }
I20260619 16:49:04.794835 16122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:04.794957 16126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:04.801494 16126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:04.810129 16126 catalog_manager.cc:1347] Generated new cluster ID: 966ac843bf3a4016b31a84853900182b
I20260619 16:49:04.810248 16126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:04.825521 16126 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:04.827489 16126 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:04.834208 16126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc: Generated new TSK 0
I20260619 16:49:04.835287 16126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:49:04.882455 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:39261
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:05.212553 16142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:05.212836 16142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:05.213008 16142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:05.220474 16142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:05.220593 16142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:05.220783 16142 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:05.220916 16142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:05.231201 16142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.153.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:05.232537 16142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:05.233927 16142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:05.247216 16147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.247159 16150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.247254 16148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:05.248035 16142 server_base.cc:1034] running on GCE node
I20260619 16:49:05.248638 16142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:05.249534 16142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:05.250784 16142 hybrid_clock.cc:648] HybridClock initialized: now 1781887745250709 us; error 43 us; skew 500 ppm
I20260619 16:49:05.251147 16142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:05 dist-test-slave-qb36 krb5kdc[16026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:05.254204 16142 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:49:05.255846 16142 webserver.cc:460] Webserver started at http://127.14.153.1:37071/ using document root <none> and password file <none>
I20260619 16:49:05.256554 16142 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:05.256685 16142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:05.257015 16142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:05.260033 16142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "52f552e28f274e41b7b74b8d0e82d54d"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-qb36"
server_key: "9d27b5334b7a9de78146d67cb54ba178"
server_key_iv: "b50e534ed8afd5e529adfc0c43e97c37"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.260888 16142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "52f552e28f274e41b7b74b8d0e82d54d"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-qb36"
server_key: "9d27b5334b7a9de78146d67cb54ba178"
server_key_iv: "b50e534ed8afd5e529adfc0c43e97c37"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.266099 16142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:49:05.269580 16157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.270888 16142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:05.271106 16142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "52f552e28f274e41b7b74b8d0e82d54d"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-qb36"
server_key: "9d27b5334b7a9de78146d67cb54ba178"
server_key_iv: "b50e534ed8afd5e529adfc0c43e97c37"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.271334 16142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:05.290400 16142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:05.294865 16142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:05.295207 16142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:05.298277 16142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:05.298405 16142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.298578 16142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:05.298660 16142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:05 dist-test-slave-qb36 krb5kdc[16026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:05.323787 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.298999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:60999 (local address 127.14.153.62:39261)
0619 16:49:05.302106 (+ 3107us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.302111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.302157 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:49:05.304553 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.304562 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.304668 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.304796 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.307400 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.308159 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.311329 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.311926 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.312118 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.317318 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.317322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.317326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.317358 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.320451 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.321248 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.321253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.321257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.321424 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.321883 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.321888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.321891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.322258 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.322414 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.323198 (+ 784us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.323422 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2993,"thread_start_us":2870,"threads_started":1}
I20260619 16:49:05.325413 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.299895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:39261 (local address 127.14.153.1:60999)
0619 16:49:05.300965 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.301028 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.304096 (+ 3068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.305144 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.305161 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.306109 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:05.307154 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.307186 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.308331 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.308339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.311132 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.311148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.312250 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.312266 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.312660 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.313496 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:05.313543 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:05.317063 (+ 3520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:05.320623 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.320633 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.320654 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.321104 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.321530 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.321535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.321539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.321753 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.322387 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:05.322396 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:05.322783 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.323082 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.323666 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":436,"threads_started":1}
I20260619 16:49:05.346110 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.331954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:39261 (local address 127.14.153.1:46071)
0619 16:49:05.333007 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.333033 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.333258 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.333631 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.333635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.334419 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:05.334756 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.334766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.336075 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.336083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.337752 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.337764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.338624 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.338630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.338805 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.339587 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:05.339605 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:05.340323 (+ 718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:05.343265 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.343271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.343275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.343905 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.344392 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.344402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.344408 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.344556 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.345278 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:05.345281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:05.345433 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.345542 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.345779 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":656,"threads_started":1}
I20260619 16:49:05.346854 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.331821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:46071 (local address 127.14.153.62:39261)
0619 16:49:05.332028 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.332034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.333134 (+ 1100us) server_negotiation.cc:407] Connection header received
0619 16:49:05.333415 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.333420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.333493 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.333603 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.334898 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.335873 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.337900 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.338414 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.338550 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.340521 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.340526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.340530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.340559 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.343126 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.344043 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.344051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.344058 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.344261 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.344672 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.344679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.344685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.345126 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.345344 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.345703 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.346010 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:49:05.362942 16142 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:34037
I20260619 16:49:05.363090 16277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:34037 every 8 connection(s)
I20260619 16:49:05.364288 16142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:05.366513 16278 heartbeater.cc:347] Connected to a master server at 127.14.153.62:39261
I20260619 16:49:05.366720 16278 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:05.367380 16278 heartbeater.cc:511] Master 127.14.153.62:39261 requested a full tablet report, sending...
I20260619 16:49:05.369264 16071 ts_manager.cc:194] Registered new tserver with Master: 52f552e28f274e41b7b74b8d0e82d54d (127.14.153.1:34037)
I20260619 16:49:05.370740 16071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:60999
I20260619 16:49:05.373315 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 16142
I20260619 16:49:05.373486 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:05.374046 14948 external_mini_cluster.cc:1428] Setting key b70d9f196150b7cdab6cfc569f618b52
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:49:05.412530 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:39261
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:05.733785 16282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:05.734153 16282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:05.734349 16282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:05.741884 16282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:05.742059 16282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:05.742228 16282 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:05.742374 16282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:05.752492 16282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.153.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:05.753808 16282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:05.755252 16282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:05.769081 16288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.769120 16290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.769456 16287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:05.770519 16282 server_base.cc:1034] running on GCE node
I20260619 16:49:05.771238 16282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:05.772181 16282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:05.773415 16282 hybrid_clock.cc:648] HybridClock initialized: now 1781887745773330 us; error 36 us; skew 500 ppm
I20260619 16:49:05.773733 16282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:05 dist-test-slave-qb36 krb5kdc[16026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:05.780869 16282 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:49:05.784601 16282 webserver.cc:460] Webserver started at http://127.14.153.2:36391/ using document root <none> and password file <none>
I20260619 16:49:05.785554 16282 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:05.785681 16282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:05.786021 16282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:05.789425 16282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "2971041890164d41b88f58e811152bdc"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-qb36"
server_key: "0092a6a1d3fe04246335ad5f36cd0810"
server_key_iv: "a127680bdf4269ddaeb32a776fc1566a"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.790271 16282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "2971041890164d41b88f58e811152bdc"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-qb36"
server_key: "0092a6a1d3fe04246335ad5f36cd0810"
server_key_iv: "a127680bdf4269ddaeb32a776fc1566a"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.795779 16282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 16:49:05.799685 16297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.801043 16282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:05.801177 16282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "2971041890164d41b88f58e811152bdc"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-qb36"
server_key: "0092a6a1d3fe04246335ad5f36cd0810"
server_key_iv: "a127680bdf4269ddaeb32a776fc1566a"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.801357 16282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:05.816330 16282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:05.820770 16282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:05.821058 16282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:05.823773 16282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:05.823892 16282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.824059 16282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:05.824138 16282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:05 dist-test-slave-qb36 krb5kdc[16026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:05.851187 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.824620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:43995 (local address 127.14.153.62:39261)
0619 16:49:05.824810 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.824817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.826271 (+ 1454us) server_negotiation.cc:407] Connection header received
0619 16:49:05.828971 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.828977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.829073 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.829181 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.832158 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.833133 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.836530 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.837516 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.837680 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.843502 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.843511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.843516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.843556 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.847257 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.848255 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.848263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.848268 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.848428 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.849069 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.849076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.849081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.849426 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.849589 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.850591 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.850815 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:49:05.852433 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.825162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:39261 (local address 127.14.153.2:43995)
0619 16:49:05.826071 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.826116 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.828631 (+ 2515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.829572 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.829591 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.830786 (+ 1195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:05.831900 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.831937 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.833358 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.833368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.836272 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.836309 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.838028 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.838033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.838335 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.839438 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:05.839479 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:05.842927 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:05.847389 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.847399 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.847417 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.848108 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.848582 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.848587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.848591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.848899 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.849575 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:05.849584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:05.850119 (+ 535us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.850488 (+ 369us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.851123 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":394,"threads_started":1}
I20260619 16:49:05.872288 16282 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:33015
I20260619 16:49:05.872310 16416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:33015 every 8 connection(s)
I20260619 16:49:05.873642 16282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:05.875916 16417 heartbeater.cc:347] Connected to a master server at 127.14.153.62:39261
I20260619 16:49:05.876150 16417 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:05.876969 16417 heartbeater.cc:511] Master 127.14.153.62:39261 requested a full tablet report, sending...
I20260619 16:49:05.878461 16071 ts_manager.cc:194] Registered new tserver with Master: 2971041890164d41b88f58e811152bdc (127.14.153.2:33015)
I20260619 16:49:05.879666 16071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:43995
I20260619 16:49:05.879951 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 16282
I20260619 16:49:05.880159 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:05.880731 14948 external_mini_cluster.cc:1428] Setting key 2ab88c8bf9d42e0e491f87751ce7223a
I20260619 16:49:05.881067 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.865915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:39261 (local address 127.14.153.2:53523)
0619 16:49:05.866499 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.866520 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.866723 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.867164 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.867168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.868051 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:05.868627 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.868646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.869986 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.869998 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.871440 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.871461 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.872544 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.872573 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.872773 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.873406 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:05.873424 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:05.874283 (+ 859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:05.878057 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.878065 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.878069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.878569 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.878907 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.878912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.878933 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.879108 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.879957 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:05.879964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:05.880155 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.880311 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.880600 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":315,"threads_started":1}
I20260619 16:49:05.881366 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.865891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:53523 (local address 127.14.153.62:39261)
0619 16:49:05.866143 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.866147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.866626 (+ 479us) server_negotiation.cc:407] Connection header received
0619 16:49:05.866874 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.866882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.866965 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.867137 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.868822 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.869848 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.871650 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.872197 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.872364 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.874476 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.874485 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.874492 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.874537 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.877913 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.878674 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.878678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.878681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.878814 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.879204 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.879207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.879209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.879742 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.880537 (+ 795us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.880671 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.880901 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:49:05.922605 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:39261
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:06.257680 16421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:06.257966 16421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:06.258080 16421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:06.265766 16421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:06.265885 16421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:06.266063 16421 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:06.266189 16421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:06.276551 16421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.153.62:39261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:06.277907 16421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:06.279417 16421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:06.293124 16427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:06.293375 16429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:06.293172 16426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:06.294168 16421 server_base.cc:1034] running on GCE node
I20260619 16:49:06.294809 16421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:06.295884 16421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:06.297132 16421 hybrid_clock.cc:648] HybridClock initialized: now 1781887746297061 us; error 48 us; skew 500 ppm
I20260619 16:49:06.297437 16421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:06 dist-test-slave-qb36 krb5kdc[16026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887746, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:06.301158 16421 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:49:06.302595 16421 webserver.cc:460] Webserver started at http://127.14.153.3:43883/ using document root <none> and password file <none>
I20260619 16:49:06.303376 16421 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:06.303490 16421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:06.303831 16421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:06.306875 16421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "3d00a1be0d7643dba8db6d752f77178c"
format_stamp: "Formatted at 2026-06-19 16:49:06 on dist-test-slave-qb36"
server_key: "03ca9b6c241f14928fddbcb2a223b877"
server_key_iv: "e3cf238d80ed14e9511b8d8c15a0e077"
server_key_version: "encryptionkey@0"
I20260619 16:49:06.307724 16421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "3d00a1be0d7643dba8db6d752f77178c"
format_stamp: "Formatted at 2026-06-19 16:49:06 on dist-test-slave-qb36"
server_key: "03ca9b6c241f14928fddbcb2a223b877"
server_key_iv: "e3cf238d80ed14e9511b8d8c15a0e077"
server_key_version: "encryptionkey@0"
I20260619 16:49:06.312779 16421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 16:49:06.316318 16436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:06.317669 16421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:49:06.317870 16421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "3d00a1be0d7643dba8db6d752f77178c"
format_stamp: "Formatted at 2026-06-19 16:49:06 on dist-test-slave-qb36"
server_key: "03ca9b6c241f14928fddbcb2a223b877"
server_key_iv: "e3cf238d80ed14e9511b8d8c15a0e077"
server_key_version: "encryptionkey@0"
I20260619 16:49:06.318113 16421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:06.347527 16421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:06.351620 16421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:06.351924 16421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:06.354754 16421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:06.354867 16421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:06.355008 16421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:06.355154 16421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 19 16:49:06 dist-test-slave-qb36 krb5kdc[16026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887746, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:06.374248 16278 heartbeater.cc:503] Master 127.14.153.62:39261 was elected leader, sending a full tablet report...
I20260619 16:49:06.379376 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.355521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:47779 (local address 127.14.153.62:39261)
0619 16:49:06.355741 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:06.355750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:06.357256 (+ 1506us) server_negotiation.cc:407] Connection header received
0619 16:49:06.359501 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:06.359506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:06.359598 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:06.359716 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:06.362041 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.363556 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.366192 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.366836 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.366954 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.372141 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:06.372146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:06.372149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:06.372188 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:06.376255 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:06.376997 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:06.377003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:06.377006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:06.377137 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:06.377574 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:06.377582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:06.377587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:06.377862 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:06.378008 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:06.378754 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 16:49:06.378976 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:49:06.380836 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.356272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:39261 (local address 127.14.153.3:47779)
0619 16:49:06.357081 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.357121 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:06.359213 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:06.360119 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:06.360138 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:06.360970 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:06.361812 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.361848 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.363749 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.363756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.365903 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.366001 (+ 98us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.367301 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.367308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.367598 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.368405 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:06.368440 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:06.371847 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:06.376394 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:06.376403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:06.376419 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:06.376864 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:06.377258 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:06.377263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:06.377267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:06.377461 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:06.377998 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:06.378006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:06.378374 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 16:49:06.378642 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 16:49:06.379253 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":354,"threads_started":1}
I20260619 16:49:06.403142 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.387998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:42929 (local address 127.14.153.62:39261)
0619 16:49:06.388191 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:06.388199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:06.388868 (+ 669us) server_negotiation.cc:407] Connection header received
0619 16:49:06.389077 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:06.389084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:06.389190 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:06.389313 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:06.391248 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.392172 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.393704 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.394308 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.394458 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.397305 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:06.397314 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:06.397322 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:06.397366 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:06.400588 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:06.401227 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:06.401235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:06.401238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:06.401362 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:06.401730 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:06.401735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:06.401739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:06.402008 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:06.402156 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:06.402532 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 16:49:06.402745 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:49:06.403273 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.388077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:39261 (local address 127.14.153.3:42929)
0619 16:49:06.388732 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.388753 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:06.388925 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:06.389538 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:06.389552 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:06.390448 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:06.391064 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.391082 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.392331 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.392340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.393561 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.393574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.394538 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.394544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.394790 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.395628 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:06.395682 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:06.397049 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:06.400714 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:06.400720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:06.400725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:06.401105 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:06.401472 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:06.401477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:06.401480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:06.401611 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:06.402124 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:06.402129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:06.402304 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 16:49:06.402442 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:49:06.402769 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":373,"threads_started":1}
I20260619 16:49:06.413028 16421 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:36271
I20260619 16:49:06.413074 16555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:36271 every 8 connection(s)
I20260619 16:49:06.414256 16421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:06.416429 16556 heartbeater.cc:347] Connected to a master server at 127.14.153.62:39261
I20260619 16:49:06.416678 16556 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:06.417408 16556 heartbeater.cc:511] Master 127.14.153.62:39261 requested a full tablet report, sending...
I20260619 16:49:06.418757 16070 ts_manager.cc:194] Registered new tserver with Master: 3d00a1be0d7643dba8db6d752f77178c (127.14.153.3:36271)
I20260619 16:49:06.419675 16070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:47779
I20260619 16:49:06.421135 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 16421
I20260619 16:49:06.421330 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:06.421849 14948 external_mini_cluster.cc:1428] Setting key 29e0b1460e353eb8a5f796988809925d
I20260619 16:49:06.424872 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:49:06 dist-test-slave-qb36 krb5kdc[16026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:49:06 dist-test-slave-qb36 krb5kdc[16026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:06.469820 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.455571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.14.153.62:39261)
0619 16:49:06.455775 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:06.455784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:06.456158 (+ 374us) server_negotiation.cc:407] Connection header received
0619 16:49:06.456381 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:06.456387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:06.456468 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:06.456577 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:06.458071 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.458793 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.460126 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.460526 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.460656 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.464558 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:06.464563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:06.464566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:06.464602 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:06.467247 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:06.467895 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:06.467900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:06.467909 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:06.468022 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:06.468408 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:06.468413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:06.468417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:06.468690 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:06.468951 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:06.469225 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 16:49:06.469441 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:49:06.473717 16070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:06.482172 16070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:06.509084 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.494456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:36271 (local address 127.0.0.1:58688)
0619 16:49:06.495595 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.495690 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:06.495856 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:06.501207 (+ 5351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:06.501214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:06.501244 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:06.501784 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.501800 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.504064 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.504074 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.506513 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.506527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.508215 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.508223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.508389 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.508428 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:49:06.508513 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:49:06.508657 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":47,"thread_start_us":7028,"threads_started":1}
I20260619 16:49:06.509222 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.495809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.14.153.2:33015)
0619 16:49:06.499556 (+ 3747us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:06.499569 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:06.501057 (+ 1488us) server_negotiation.cc:407] Connection header received
0619 16:49:06.501396 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:06.501403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:06.501630 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:06.501849 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:06.502418 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.503739 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.506309 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.507618 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.507772 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.507984 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:06.508780 (+ 796us) server_negotiation.cc:299] Negotiation successful
0619 16:49:06.508949 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3577,"thread_start_us":3476,"threads_started":1}
I20260619 16:49:06.509795 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.494810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:33015 (local address 127.0.0.1:46376)
0619 16:49:06.500908 (+ 6098us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.500934 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:06.501183 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:06.501857 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:06.501864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:06.501889 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:06.502274 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.502283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.503902 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.503907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.506140 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.506158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.507858 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.507865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.508059 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.508125 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 16:49:06.508263 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:49:06.508488 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5933,"mutex_wait_us":46,"thread_start_us":2502,"threads_started":1}
I20260619 16:49:06.510499 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.494982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:34037 (local address 127.0.0.1:35800)
0619 16:49:06.501783 (+ 6801us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.501802 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:06.501999 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:06.502593 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:06.502597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:06.502621 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:06.503054 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.503066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.504953 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.504961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.507067 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.507081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.509686 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.509693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.509905 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.509937 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:49:06.510058 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:49:06.510185 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6659,"thread_start_us":5582,"threads_started":1}
I20260619 16:49:06.510511 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.494688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.14.153.3:36271)
0619 16:49:06.500523 (+ 5835us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:06.500534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:06.500568 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:49:06.500682 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:06.500691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:06.500964 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:06.501993 (+ 1029us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:06.502286 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.503915 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.506691 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.507977 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.508104 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.508324 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:06.510013 (+ 1689us) server_negotiation.cc:299] Negotiation successful
0619 16:49:06.510186 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5651,"thread_start_us":5474,"threads_started":1}
I20260619 16:49:06.510555 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.495455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.14.153.1:34037)
0619 16:49:06.499978 (+ 4523us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:06.499985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:06.501899 (+ 1914us) server_negotiation.cc:407] Connection header received
0619 16:49:06.502177 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:06.502184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:06.502398 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:06.502600 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:06.503225 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.504748 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.507228 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.509185 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.509347 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.509545 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:06.510129 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 16:49:06.510300 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4387,"thread_start_us":4264,"threads_started":1}
I20260619 16:49:06.513536 16347 tablet_service.cc:1467] Processing CreateTablet for tablet 8e79846b90b547929a0af94146054b79 (DEFAULT_TABLE table=test-table [id=abe8f8a8c9d64e07a4316c31fc2a2a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:06.514001 16486 tablet_service.cc:1467] Processing CreateTablet for tablet 8e79846b90b547929a0af94146054b79 (DEFAULT_TABLE table=test-table [id=abe8f8a8c9d64e07a4316c31fc2a2a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:06.514001 16212 tablet_service.cc:1467] Processing CreateTablet for tablet 8e79846b90b547929a0af94146054b79 (DEFAULT_TABLE table=test-table [id=abe8f8a8c9d64e07a4316c31fc2a2a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:06.514958 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e79846b90b547929a0af94146054b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:06.515398 16212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e79846b90b547929a0af94146054b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:06.515398 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e79846b90b547929a0af94146054b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:06.527297 16575 tablet_bootstrap.cc:492] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: Bootstrap starting.
I20260619 16:49:06.529681 16576 tablet_bootstrap.cc:492] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: Bootstrap starting.
I20260619 16:49:06.531270 16575 tablet_bootstrap.cc:654] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:06.531800 16577 tablet_bootstrap.cc:492] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: Bootstrap starting.
I20260619 16:49:06.532704 16575 log.cc:824] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:06.533476 16576 tablet_bootstrap.cc:654] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:06.534739 16576 log.cc:824] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:06.535071 16577 tablet_bootstrap.cc:654] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:06.536116 16575 tablet_bootstrap.cc:492] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: No bootstrap required, opened a new log
I20260619 16:49:06.536214 16577 log.cc:824] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:06.536542 16575 ts_tablet_manager.cc:1397] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260619 16:49:06.542819 16577 tablet_bootstrap.cc:492] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: No bootstrap required, opened a new log
I20260619 16:49:06.543236 16577 ts_tablet_manager.cc:1397] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20260619 16:49:06.545620 16576 tablet_bootstrap.cc:492] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: No bootstrap required, opened a new log
I20260619 16:49:06.545895 16576 ts_tablet_manager.cc:1397] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260619 16:49:06.545915 16575 raft_consensus.cc:348] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } }
I20260619 16:49:06.546331 16575 raft_consensus.cc:374] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:06.546434 16575 raft_consensus.cc:729] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2971041890164d41b88f58e811152bdc, State: Initialized, Role: FOLLOWER
I20260619 16:49:06.547194 16575 consensus_queue.cc:260] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } }
I20260619 16:49:06.548373 16417 heartbeater.cc:503] Master 127.14.153.62:39261 was elected leader, sending a full tablet report...
I20260619 16:49:06.549062 16575 ts_tablet_manager.cc:1428] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 16:49:06.554183 16577 raft_consensus.cc:348] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } }
I20260619 16:49:06.554616 16577 raft_consensus.cc:374] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:06.554733 16577 raft_consensus.cc:729] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d00a1be0d7643dba8db6d752f77178c, State: Initialized, Role: FOLLOWER
I20260619 16:49:06.555393 16577 consensus_queue.cc:260] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } }
I20260619 16:49:06.556488 16556 heartbeater.cc:503] Master 127.14.153.62:39261 was elected leader, sending a full tablet report...
I20260619 16:49:06.557233 16577 ts_tablet_manager.cc:1428] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 16:49:06.557194 16576 raft_consensus.cc:348] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } }
I20260619 16:49:06.557684 16576 raft_consensus.cc:374] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:06.557808 16576 raft_consensus.cc:729] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f552e28f274e41b7b74b8d0e82d54d, State: Initialized, Role: FOLLOWER
I20260619 16:49:06.558780 16576 consensus_queue.cc:260] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } }
I20260619 16:49:06.560565 16576 ts_tablet_manager.cc:1428] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260619 16:49:06.616850 16279 tablet.cc:2369] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:06.625811 16418 tablet.cc:2369] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:06.665951 16557 tablet.cc:2369] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:06.800179 16582 raft_consensus.cc:482] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:06.800488 16582 raft_consensus.cc:504] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } }
I20260619 16:49:06.802189 16582 leader_election.cc:288] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f552e28f274e41b7b74b8d0e82d54d (127.14.153.1:34037), 2971041890164d41b88f58e811152bdc (127.14.153.2:33015)
I20260619 16:49:06.808900 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.802591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:33015 (local address 127.14.153.3:46317)
0619 16:49:06.802766 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.802788 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:06.802904 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:06.803297 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:06.803302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:06.803329 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:06.803551 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.803561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.805264 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.805271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.807112 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.807124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.808178 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.808184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.808358 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.808382 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:49:06.808487 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:49:06.808615 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 16:49:06.808895 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.802658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:46317 (local address 127.14.153.2:33015)
0619 16:49:06.802835 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:06.802841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:06.802864 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:49:06.803080 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:06.803087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:06.803158 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:06.803270 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:06.803706 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.805128 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.807265 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.808011 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.808183 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.808256 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:06.808512 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 16:49:06.808638 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:49:06.809930 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.802591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:34037 (local address 127.14.153.3:51597)
0619 16:49:06.803706 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.803726 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:06.804033 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:06.804427 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:06.804430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:06.804449 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:06.805256 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.805265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.806648 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.806652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.807996 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:06.808008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.809262 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.809268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:06.809437 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.809474 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:49:06.809569 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:49:06.809666 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":386,"thread_start_us":401,"threads_started":1}
I20260619 16:49:06.809998 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.802609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:51597 (local address 127.14.153.1:34037)
0619 16:49:06.802854 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:06.802861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:06.803972 (+ 1111us) server_negotiation.cc:407] Connection header received
0619 16:49:06.804184 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:06.804189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:06.804272 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:06.804364 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:06.805406 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.806505 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.808157 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.808914 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.809109 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.809180 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:06.809612 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 16:49:06.809756 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 16:49:06.810020 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79846b90b547929a0af94146054b79" candidate_uuid: "3d00a1be0d7643dba8db6d752f77178c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2971041890164d41b88f58e811152bdc" is_pre_election: true
I20260619 16:49:06.810637 16367 raft_consensus.cc:2454] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d00a1be0d7643dba8db6d752f77178c in term 0.
I20260619 16:49:06.810935 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79846b90b547929a0af94146054b79" candidate_uuid: "3d00a1be0d7643dba8db6d752f77178c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f552e28f274e41b7b74b8d0e82d54d" is_pre_election: true
I20260619 16:49:06.811338 16438 leader_election.cc:302] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2971041890164d41b88f58e811152bdc, 3d00a1be0d7643dba8db6d752f77178c; no voters:
I20260619 16:49:06.811532 16233 raft_consensus.cc:2454] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d00a1be0d7643dba8db6d752f77178c in term 0.
I20260619 16:49:06.811730 16582 raft_consensus.cc:2781] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:06.811877 16582 raft_consensus.cc:482] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:06.812011 16582 raft_consensus.cc:3037] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:06.814204 16582 raft_consensus.cc:504] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } }
I20260619 16:49:06.814798 16582 leader_election.cc:288] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [CANDIDATE]: Term 1 election: Requested vote from peers 52f552e28f274e41b7b74b8d0e82d54d (127.14.153.1:34037), 2971041890164d41b88f58e811152bdc (127.14.153.2:33015)
I20260619 16:49:06.815171 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79846b90b547929a0af94146054b79" candidate_uuid: "3d00a1be0d7643dba8db6d752f77178c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f552e28f274e41b7b74b8d0e82d54d"
I20260619 16:49:06.815361 16233 raft_consensus.cc:3037] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:06.815346 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e79846b90b547929a0af94146054b79" candidate_uuid: "3d00a1be0d7643dba8db6d752f77178c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2971041890164d41b88f58e811152bdc"
I20260619 16:49:06.815557 16367 raft_consensus.cc:3037] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:06.817881 16233 raft_consensus.cc:2454] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d00a1be0d7643dba8db6d752f77178c in term 1.
I20260619 16:49:06.818208 16367 raft_consensus.cc:2454] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d00a1be0d7643dba8db6d752f77178c in term 1.
I20260619 16:49:06.818289 16440 leader_election.cc:302] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d00a1be0d7643dba8db6d752f77178c, 52f552e28f274e41b7b74b8d0e82d54d; no voters:
I20260619 16:49:06.818612 16582 raft_consensus.cc:2781] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:06.819000 16582 raft_consensus.cc:686] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 1 LEADER]: Becoming Leader. State: Replica: 3d00a1be0d7643dba8db6d752f77178c, State: Running, Role: LEADER
I20260619 16:49:06.819594 16582 consensus_queue.cc:237] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } }
I20260619 16:49:06.824293 16071 catalog_manager.cc:5495] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d00a1be0d7643dba8db6d752f77178c (127.14.153.3). New cstate: current_term: 1 leader_uuid: "3d00a1be0d7643dba8db6d752f77178c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:06.881038 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.874567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.14.153.3:36271)
0619 16:49:06.874772 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:06.874778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:06.874801 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:49:06.875151 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:06.875157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:06.875263 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:06.875431 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:06.875747 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.877199 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.878506 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:06.879166 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:06.879286 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:06.879795 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:06.880452 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:06.880598 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:06.880682 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 16:49:06.880771 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:49:06.889216 16367 raft_consensus.cc:1261] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [term 1 FOLLOWER]: Refusing update from remote peer 3d00a1be0d7643dba8db6d752f77178c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:06.889240 16233 raft_consensus.cc:1261] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Refusing update from remote peer 3d00a1be0d7643dba8db6d752f77178c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:06.890192 16585 consensus_queue.cc:1035] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:06.890614 16582 consensus_queue.cc:1035] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:06.903240 16590 mvcc.cc:205] Tried to move back new op lower bound from 7298612211248689152 to 7298612210977136640. Current Snapshot: MvccSnapshot[applied={T|T < 7298612211248689152}]
I20260619 16:49:06.906543 16588 mvcc.cc:205] Tried to move back new op lower bound from 7298612211248689152 to 7298612210977136640. Current Snapshot: MvccSnapshot[applied={T|T < 7298612211248689152}]
I20260619 16:49:06.908528 16589 mvcc.cc:205] Tried to move back new op lower bound from 7298612211248689152 to 7298612210977136640. Current Snapshot: MvccSnapshot[applied={T|T < 7298612211248689152}]
I20260619 16:49:06.924294 16071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:06.924571 16071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:06.928126 16071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc: Sending DeleteTablet for 3 replicas of tablet 8e79846b90b547929a0af94146054b79
I20260619 16:49:06.929322 16486 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79846b90b547929a0af94146054b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:06 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:58688
I20260619 16:49:06.929387 16212 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79846b90b547929a0af94146054b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:06 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:35800
I20260619 16:49:06.930145 16347 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e79846b90b547929a0af94146054b79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:06 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:46376
I20260619 16:49:06.932823 16071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:06.934015 16071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:06.944525 16211 tablet_service.cc:1467] Processing CreateTablet for tablet 49172d81c08a4fc7b54dbc8d0665e537 (DEFAULT_TABLE table=test-table [id=0abb465d6cbc459991361f2c1fb47510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:06.944542 16346 tablet_service.cc:1467] Processing CreateTablet for tablet 49172d81c08a4fc7b54dbc8d0665e537 (DEFAULT_TABLE table=test-table [id=0abb465d6cbc459991361f2c1fb47510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:06.945159 16211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49172d81c08a4fc7b54dbc8d0665e537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:06.945250 16346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49172d81c08a4fc7b54dbc8d0665e537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:06.948432 16485 tablet_service.cc:1467] Processing CreateTablet for tablet 49172d81c08a4fc7b54dbc8d0665e537 (DEFAULT_TABLE table=test-table [id=0abb465d6cbc459991361f2c1fb47510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:06.948979 16485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49172d81c08a4fc7b54dbc8d0665e537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:06.953111 16576 tablet_bootstrap.cc:492] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: Bootstrap starting.
I20260619 16:49:06.955479 16576 tablet_bootstrap.cc:654] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:06.957158 16577 tablet_bootstrap.cc:492] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: Bootstrap starting.
I20260619 16:49:06.959703 16577 tablet_bootstrap.cc:654] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:06.961843 16575 tablet_bootstrap.cc:492] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: Bootstrap starting.
I20260619 16:49:06.963819 16575 tablet_bootstrap.cc:654] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:06.975617 16601 tablet_replica.cc:330] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: stopping tablet replica
I20260619 16:49:06.976871 16601 raft_consensus.cc:2229] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:06.977725 16600 tablet_replica.cc:330] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: stopping tablet replica
I20260619 16:49:06.978140 16600 raft_consensus.cc:2229] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:06.978724 16600 raft_consensus.cc:2258] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:06.979808 16601 raft_consensus.cc:2258] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:06.982419 16601 ts_tablet_manager.cc:1905] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:06.984673 16600 ts_tablet_manager.cc:1905] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:06.985710 16575 tablet_bootstrap.cc:492] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: No bootstrap required, opened a new log
I20260619 16:49:06.986042 16575 ts_tablet_manager.cc:1397] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.001s
I20260619 16:49:06.987272 16575 raft_consensus.cc:348] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } }
I20260619 16:49:06.988049 16575 raft_consensus.cc:374] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:06.988243 16575 raft_consensus.cc:729] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2971041890164d41b88f58e811152bdc, State: Initialized, Role: FOLLOWER
I20260619 16:49:06.990248 16575 consensus_queue.cc:260] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } }
I20260619 16:49:06.992311 16601 ts_tablet_manager.cc:1918] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:06.992314 16600 ts_tablet_manager.cc:1918] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:06.992475 16600 log.cc:1196] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/8e79846b90b547929a0af94146054b79
I20260619 16:49:06.992506 16601 log.cc:1196] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/8e79846b90b547929a0af94146054b79
I20260619 16:49:06.993260 16601 ts_tablet_manager.cc:1939] T 8e79846b90b547929a0af94146054b79 P 2971041890164d41b88f58e811152bdc: Deleting consensus metadata
I20260619 16:49:06.993924 16576 tablet_bootstrap.cc:492] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: No bootstrap required, opened a new log
I20260619 16:49:06.994123 16576 ts_tablet_manager.cc:1397] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.006s
I20260619 16:49:06.994624 16575 ts_tablet_manager.cc:1428] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260619 16:49:06.995057 16576 raft_consensus.cc:348] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } }
I20260619 16:49:06.995384 16576 raft_consensus.cc:374] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:06.995537 16576 raft_consensus.cc:729] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f552e28f274e41b7b74b8d0e82d54d, State: Initialized, Role: FOLLOWER
I20260619 16:49:06.995910 16576 consensus_queue.cc:260] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } }
I20260619 16:49:06.996177 16056 catalog_manager.cc:4841] TS 2971041890164d41b88f58e811152bdc (127.14.153.2:33015): tablet 8e79846b90b547929a0af94146054b79 (table test-table [id=abe8f8a8c9d64e07a4316c31fc2a2a15]) successfully deleted
I20260619 16:49:06.996578 16576 ts_tablet_manager.cc:1428] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:06.997177 16600 ts_tablet_manager.cc:1939] T 8e79846b90b547929a0af94146054b79 P 3d00a1be0d7643dba8db6d752f77178c: Deleting consensus metadata
I20260619 16:49:06.998833 16057 catalog_manager.cc:4841] TS 3d00a1be0d7643dba8db6d752f77178c (127.14.153.3:36271): tablet 8e79846b90b547929a0af94146054b79 (table test-table [id=abe8f8a8c9d64e07a4316c31fc2a2a15]) successfully deleted
I20260619 16:49:06.999441 16577 tablet_bootstrap.cc:492] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: No bootstrap required, opened a new log
I20260619 16:49:06.999580 16577 ts_tablet_manager.cc:1397] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.004s
I20260619 16:49:07.000427 16577 raft_consensus.cc:348] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } }
I20260619 16:49:07.000638 16577 raft_consensus.cc:374] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:07.000739 16577 raft_consensus.cc:729] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d00a1be0d7643dba8db6d752f77178c, State: Initialized, Role: FOLLOWER
I20260619 16:49:07.001008 16577 consensus_queue.cc:260] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } }
I20260619 16:49:07.001505 16577 ts_tablet_manager.cc:1428] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 16:49:07.003638 16599 tablet_replica.cc:330] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: stopping tablet replica
I20260619 16:49:07.004591 16599 raft_consensus.cc:2229] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:07.005635 16599 raft_consensus.cc:2258] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:07.007620 16599 ts_tablet_manager.cc:1905] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:07.010088 16581 raft_consensus.cc:482] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:07.010324 16581 raft_consensus.cc:504] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } }
I20260619 16:49:07.012256 16581 leader_election.cc:288] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f552e28f274e41b7b74b8d0e82d54d (127.14.153.1:34037), 3d00a1be0d7643dba8db6d752f77178c (127.14.153.3:36271)
I20260619 16:49:07.015527 16599 ts_tablet_manager.cc:1918] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:07.015745 16599 log.cc:1196] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/8e79846b90b547929a0af94146054b79
I20260619 16:49:07.016425 16599 ts_tablet_manager.cc:1939] T 8e79846b90b547929a0af94146054b79 P 52f552e28f274e41b7b74b8d0e82d54d: Deleting consensus metadata
I20260619 16:49:07.018632 16058 catalog_manager.cc:4841] TS 52f552e28f274e41b7b74b8d0e82d54d (127.14.153.1:34037): tablet 8e79846b90b547929a0af94146054b79 (table test-table [id=abe8f8a8c9d64e07a4316c31fc2a2a15]) successfully deleted
I20260619 16:49:07.025482 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.013199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:38295 (local address 127.14.153.1:34037)
0619 16:49:07.013422 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.013430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.019385 (+ 5955us) server_negotiation.cc:407] Connection header received
0619 16:49:07.019644 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.019650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.019746 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.019827 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:07.020318 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.021109 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.023655 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.024431 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.024622 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.024701 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.025022 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.025151 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:49:07.025695 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.012825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:34037 (local address 127.14.153.2:38295)
0619 16:49:07.019263 (+ 6438us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.019290 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:07.019499 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:07.019849 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:07.019853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:07.019880 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:07.020157 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.020165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.021245 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.021250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.023504 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.023517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.024662 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.024668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.024854 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.024880 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:49:07.024999 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:49:07.025137 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6227,"mutex_wait_us":46,"thread_start_us":6435,"threads_started":1}
I20260619 16:49:07.026314 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49172d81c08a4fc7b54dbc8d0665e537" candidate_uuid: "2971041890164d41b88f58e811152bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f552e28f274e41b7b74b8d0e82d54d" is_pre_election: true
I20260619 16:49:07.026511 16233 raft_consensus.cc:2454] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2971041890164d41b88f58e811152bdc in term 0.
I20260619 16:49:07.026974 16301 leader_election.cc:302] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2971041890164d41b88f58e811152bdc, 52f552e28f274e41b7b74b8d0e82d54d; no voters:
I20260619 16:49:07.027496 16581 raft_consensus.cc:2781] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:07.027649 16581 raft_consensus.cc:482] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:07.027750 16581 raft_consensus.cc:3037] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:07.028537 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.012839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:55059 (local address 127.14.153.3:36271)
0619 16:49:07.013023 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.013031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.019703 (+ 6672us) server_negotiation.cc:407] Connection header received
0619 16:49:07.019931 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.019937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.020015 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.020081 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:07.021407 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.022600 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.025245 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.026636 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.026762 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.026847 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.027621 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.027791 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:49:07.028548 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.012915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:36271 (local address 127.14.153.2:55059)
0619 16:49:07.019601 (+ 6686us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.019620 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:07.019748 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:07.020283 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:07.020287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:07.020314 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:07.021153 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.021167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.022749 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.022754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.025029 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.025048 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.027234 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.027241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.027426 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.027462 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:49:07.027573 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:49:07.027731 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6533,"mutex_wait_us":7,"thread_start_us":5950,"threads_started":1}
I20260619 16:49:07.029419 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49172d81c08a4fc7b54dbc8d0665e537" candidate_uuid: "2971041890164d41b88f58e811152bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d00a1be0d7643dba8db6d752f77178c" is_pre_election: true
I20260619 16:49:07.029902 16511 raft_consensus.cc:2454] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2971041890164d41b88f58e811152bdc in term 0.
I20260619 16:49:07.030779 16581 raft_consensus.cc:504] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } }
I20260619 16:49:07.031526 16581 leader_election.cc:288] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [CANDIDATE]: Term 1 election: Requested vote from peers 52f552e28f274e41b7b74b8d0e82d54d (127.14.153.1:34037), 3d00a1be0d7643dba8db6d752f77178c (127.14.153.3:36271)
I20260619 16:49:07.031957 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49172d81c08a4fc7b54dbc8d0665e537" candidate_uuid: "2971041890164d41b88f58e811152bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f552e28f274e41b7b74b8d0e82d54d"
I20260619 16:49:07.032039 16511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49172d81c08a4fc7b54dbc8d0665e537" candidate_uuid: "2971041890164d41b88f58e811152bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d00a1be0d7643dba8db6d752f77178c"
I20260619 16:49:07.032160 16233 raft_consensus.cc:3037] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:07.032261 16511 raft_consensus.cc:3037] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:07.034224 16233 raft_consensus.cc:2454] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2971041890164d41b88f58e811152bdc in term 1.
I20260619 16:49:07.034622 16301 leader_election.cc:302] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2971041890164d41b88f58e811152bdc, 52f552e28f274e41b7b74b8d0e82d54d; no voters:
I20260619 16:49:07.034919 16581 raft_consensus.cc:2781] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:07.035511 16581 raft_consensus.cc:686] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 1 LEADER]: Becoming Leader. State: Replica: 2971041890164d41b88f58e811152bdc, State: Running, Role: LEADER
I20260619 16:49:07.035573 16511 raft_consensus.cc:2454] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2971041890164d41b88f58e811152bdc in term 1.
I20260619 16:49:07.035975 16581 consensus_queue.cc:237] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } }
I20260619 16:49:07.039716 16071 catalog_manager.cc:5495] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2971041890164d41b88f58e811152bdc (127.14.153.2). New cstate: current_term: 1 leader_uuid: "2971041890164d41b88f58e811152bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:49:07 dist-test-slave-qb36 krb5kdc[16026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887744, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:07.084694 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.069473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.14.153.62:39261)
0619 16:49:07.070093 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.070101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.070445 (+ 344us) server_negotiation.cc:407] Connection header received
0619 16:49:07.070596 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.070602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.070680 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.070804 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:07.072250 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.073414 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.074644 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.075311 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.075464 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.078980 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:07.078985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:07.078989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:07.079016 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:07.081651 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.082507 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.082532 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.082538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.082674 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.083105 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.083109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.083111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.083416 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:07.083538 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.084080 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.084389 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":468,"thread_start_us":305,"threads_started":1}
I20260619 16:49:07.084848 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.069466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:39261 (local address 127.0.0.1:50414)
0619 16:49:07.070304 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.070339 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:07.070471 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:07.070836 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:07.070840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:07.071907 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:07.072091 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.072100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.073558 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.073562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.074517 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.074527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.075492 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.075502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.075617 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.076386 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:07.076402 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:07.078782 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:07.081780 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:07.081794 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:07.081803 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:07.082389 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:07.082781 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:07.082786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:07.082790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:07.082971 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:07.083543 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:07.083552 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:07.083745 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 16:49:07.083962 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 16:49:07.084274 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":549,"threads_started":1}
I20260619 16:49:07.090499 16071 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:50414:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:49:07.091555 16071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:07.098392 16347 tablet_service.cc:1467] Processing CreateTablet for tablet e37ea2cbdf0841d488a3103670d9900e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b30c5f46b66948a0ba31696e9ba33230]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:07.098951 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37ea2cbdf0841d488a3103670d9900e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:07.098825 16212 tablet_service.cc:1467] Processing CreateTablet for tablet e37ea2cbdf0841d488a3103670d9900e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b30c5f46b66948a0ba31696e9ba33230]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:07.099401 16212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37ea2cbdf0841d488a3103670d9900e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:07.103839 16486 tablet_service.cc:1467] Processing CreateTablet for tablet e37ea2cbdf0841d488a3103670d9900e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b30c5f46b66948a0ba31696e9ba33230]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:07.104400 16486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37ea2cbdf0841d488a3103670d9900e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:07.104425 16575 tablet_bootstrap.cc:492] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: Bootstrap starting.
I20260619 16:49:07.106061 16575 tablet_bootstrap.cc:654] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:07.107241 16576 tablet_bootstrap.cc:492] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: Bootstrap starting.
I20260619 16:49:07.108821 16576 tablet_bootstrap.cc:654] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:07.111364 16576 tablet_bootstrap.cc:492] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: No bootstrap required, opened a new log
I20260619 16:49:07.111574 16576 ts_tablet_manager.cc:1397] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:49:07.112323 16576 raft_consensus.cc:348] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } }
I20260619 16:49:07.112653 16576 raft_consensus.cc:374] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:07.112768 16576 raft_consensus.cc:729] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f552e28f274e41b7b74b8d0e82d54d, State: Initialized, Role: FOLLOWER
I20260619 16:49:07.112857 16575 tablet_bootstrap.cc:492] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: No bootstrap required, opened a new log
I20260619 16:49:07.113036 16575 ts_tablet_manager.cc:1397] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260619 16:49:07.113174 16576 consensus_queue.cc:260] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } }
I20260619 16:49:07.113770 16575 raft_consensus.cc:348] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } }
I20260619 16:49:07.113690 16583 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.114004 16575 raft_consensus.cc:374] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:07.114030 16583 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:07.114080 16575 raft_consensus.cc:729] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2971041890164d41b88f58e811152bdc, State: Initialized, Role: FOLLOWER
I20260619 16:49:07.114337 16575 consensus_queue.cc:260] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } }
I20260619 16:49:07.114735 16581 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.114986 16581 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:07.115425 16577 tablet_bootstrap.cc:492] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: Bootstrap starting.
I20260619 16:49:07.116178 16576 ts_tablet_manager.cc:1428] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260619 16:49:07.116647 16575 ts_tablet_manager.cc:1428] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260619 16:49:07.117138 16577 tablet_bootstrap.cc:654] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:07.120162 16577 tablet_bootstrap.cc:492] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: No bootstrap required, opened a new log
I20260619 16:49:07.120368 16577 ts_tablet_manager.cc:1397] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 16:49:07.121197 16577 raft_consensus.cc:348] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } }
I20260619 16:49:07.121505 16577 raft_consensus.cc:374] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:07.121647 16577 raft_consensus.cc:729] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d00a1be0d7643dba8db6d752f77178c, State: Initialized, Role: FOLLOWER
I20260619 16:49:07.121906 16577 consensus_queue.cc:260] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } }
I20260619 16:49:07.122510 16577 ts_tablet_manager.cc:1428] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 16:49:07.122363 16582 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.122658 16582 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:07.170298 16582 raft_consensus.cc:482] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:07.170526 16582 raft_consensus.cc:504] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } }
I20260619 16:49:07.171316 16582 leader_election.cc:288] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2971041890164d41b88f58e811152bdc (127.14.153.2:33015), 52f552e28f274e41b7b74b8d0e82d54d (127.14.153.1:34037)
I20260619 16:49:07.171828 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37ea2cbdf0841d488a3103670d9900e" candidate_uuid: "3d00a1be0d7643dba8db6d752f77178c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2971041890164d41b88f58e811152bdc" is_pre_election: true
I20260619 16:49:07.171967 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37ea2cbdf0841d488a3103670d9900e" candidate_uuid: "3d00a1be0d7643dba8db6d752f77178c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f552e28f274e41b7b74b8d0e82d54d" is_pre_election: true
I20260619 16:49:07.172070 16367 raft_consensus.cc:2454] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d00a1be0d7643dba8db6d752f77178c in term 0.
I20260619 16:49:07.172202 16233 raft_consensus.cc:2454] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d00a1be0d7643dba8db6d752f77178c in term 0.
I20260619 16:49:07.172550 16438 leader_election.cc:302] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2971041890164d41b88f58e811152bdc, 3d00a1be0d7643dba8db6d752f77178c; no voters:
I20260619 16:49:07.172839 16582 raft_consensus.cc:2781] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:07.172991 16582 raft_consensus.cc:482] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:07.173110 16582 raft_consensus.cc:3037] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:07.175626 16582 raft_consensus.cc:504] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } }
I20260619 16:49:07.176268 16582 leader_election.cc:288] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [CANDIDATE]: Term 1 election: Requested vote from peers 2971041890164d41b88f58e811152bdc (127.14.153.2:33015), 52f552e28f274e41b7b74b8d0e82d54d (127.14.153.1:34037)
I20260619 16:49:07.176730 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37ea2cbdf0841d488a3103670d9900e" candidate_uuid: "3d00a1be0d7643dba8db6d752f77178c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2971041890164d41b88f58e811152bdc"
I20260619 16:49:07.176782 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37ea2cbdf0841d488a3103670d9900e" candidate_uuid: "3d00a1be0d7643dba8db6d752f77178c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f552e28f274e41b7b74b8d0e82d54d"
I20260619 16:49:07.176935 16367 raft_consensus.cc:3037] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:07.177019 16233 raft_consensus.cc:3037] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:07.179476 16367 raft_consensus.cc:2454] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d00a1be0d7643dba8db6d752f77178c in term 1.
I20260619 16:49:07.179934 16438 leader_election.cc:302] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2971041890164d41b88f58e811152bdc, 3d00a1be0d7643dba8db6d752f77178c; no voters:
I20260619 16:49:07.179926 16233 raft_consensus.cc:2454] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d00a1be0d7643dba8db6d752f77178c in term 1.
I20260619 16:49:07.180265 16582 raft_consensus.cc:2781] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:07.180452 16582 raft_consensus.cc:686] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [term 1 LEADER]: Becoming Leader. State: Replica: 3d00a1be0d7643dba8db6d752f77178c, State: Running, Role: LEADER
I20260619 16:49:07.180733 16582 consensus_queue.cc:237] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } }
I20260619 16:49:07.182426 16597 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: TxnStatusTablet state changed. Reason: New leader 3d00a1be0d7643dba8db6d752f77178c. Latest consensus state: current_term: 1 leader_uuid: "3d00a1be0d7643dba8db6d752f77178c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.182826 16597 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:07.183699 16620 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:49:07.183961 16620 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:49:07.184299 16070 catalog_manager.cc:5495] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d00a1be0d7643dba8db6d752f77178c (127.14.153.3). New cstate: current_term: 1 leader_uuid: "3d00a1be0d7643dba8db6d752f77178c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:07.200997 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.196078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.14.153.3:36271)
0619 16:49:07.196256 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.196262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.196282 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:49:07.196386 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.196391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.196474 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.196600 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:07.196942 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.197839 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.199221 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.199679 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.199817 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.200255 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:07.200366 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:07.200463 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.200611 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.200684 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:49:07.201014 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.195722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:36271 (local address 127.0.0.1:58696)
0619 16:49:07.195953 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.195975 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:07.196109 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:07.196608 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:07.196613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:07.196627 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:07.196786 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.196793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.197958 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.197965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.199070 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.199086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.199847 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.199853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.200026 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.200079 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:07.200460 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:07.200482 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:49:07.200583 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:49:07.200728 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:49:07.204577 16233 raft_consensus.cc:1261] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Refusing update from remote peer 3d00a1be0d7643dba8db6d752f77178c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:07.204631 16367 raft_consensus.cc:1261] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc [term 1 FOLLOWER]: Refusing update from remote peer 3d00a1be0d7643dba8db6d752f77178c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:07.205156 16597 consensus_queue.cc:1035] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:07.205524 16582 consensus_queue.cc:1035] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:07.208820 16581 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: TxnStatusTablet state changed. Reason: New leader 3d00a1be0d7643dba8db6d752f77178c. Latest consensus state: current_term: 1 leader_uuid: "3d00a1be0d7643dba8db6d752f77178c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.209107 16581 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:07.210132 16583 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: TxnStatusTablet state changed. Reason: New leader 3d00a1be0d7643dba8db6d752f77178c. Latest consensus state: current_term: 1 leader_uuid: "3d00a1be0d7643dba8db6d752f77178c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.210397 16583 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:07.212683 16595 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d00a1be0d7643dba8db6d752f77178c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.213060 16595 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:07.213299 16595 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d00a1be0d7643dba8db6d752f77178c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.213517 16595 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 3d00a1be0d7643dba8db6d752f77178c: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:07.214591 16583 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d00a1be0d7643dba8db6d752f77178c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.214792 16583 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 52f552e28f274e41b7b74b8d0e82d54d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:07.217381 16581 tablet_replica.cc:439] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d00a1be0d7643dba8db6d752f77178c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2971041890164d41b88f58e811152bdc" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 33015 } } peers { permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 } } peers { permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 } } }
I20260619 16:49:07.217731 16581 tablet_replica.cc:442] T e37ea2cbdf0841d488a3103670d9900e P 2971041890164d41b88f58e811152bdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:07.227236 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.220046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.14.153.2:33015)
0619 16:49:07.220601 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.220609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.221019 (+ 410us) server_negotiation.cc:407] Connection header received
0619 16:49:07.221317 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.221322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.221440 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.221600 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:07.222051 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.222954 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.224830 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.225273 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.225378 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.225967 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:07.226534 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:07.226644 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.226793 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.226869 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434}
I20260619 16:49:07.244599 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.238381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:47685 (local address 127.14.153.3:36271)
0619 16:49:07.238727 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.238731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.239604 (+ 873us) server_negotiation.cc:407] Connection header received
0619 16:49:07.239830 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.239834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.239903 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.240002 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:07.240462 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.241212 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.242566 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.243159 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.243318 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.243750 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:07.243894 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:07.244030 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.244175 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.244267 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:49:07.244692 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.238793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:36271 (local address 127.14.153.2:47685)
0619 16:49:07.239500 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.239517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:07.239681 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:07.240017 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:07.240021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:07.240033 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:07.240304 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.240312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.241351 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.241358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.242425 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.242437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.243347 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.243354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.243506 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.243562 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:07.244045 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:07.244064 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:49:07.244154 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:49:07.244288 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":409,"threads_started":1}
I20260619 16:49:07.254706 16511 raft_consensus.cc:1261] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [term 1 FOLLOWER]: Refusing update from remote peer 2971041890164d41b88f58e811152bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:07.255308 16233 raft_consensus.cc:1261] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Refusing update from remote peer 2971041890164d41b88f58e811152bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:07.255972 16607 consensus_queue.cc:1035] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f552e28f274e41b7b74b8d0e82d54d" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:07.256609 16581 consensus_queue.cc:1035] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d00a1be0d7643dba8db6d752f77178c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:07.294162 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.286871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:45067 (local address 127.14.153.2:33015)
0619 16:49:07.287067 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.287074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.287848 (+ 774us) server_negotiation.cc:407] Connection header received
0619 16:49:07.288154 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.288160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.288247 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.288348 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:07.289336 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.290375 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.292139 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.292733 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.292863 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.293465 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:07.293598 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:07.293671 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.293893 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.293967 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:49:07.294435 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.286919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:33015 (local address 127.14.153.3:45067)
0619 16:49:07.287750 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.287766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:07.287921 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:07.288370 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:07.288373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:07.288383 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:07.289142 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.289160 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.290520 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.290524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.291984 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.291999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.292946 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.292955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.293173 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.293245 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:07.293770 (+ 525us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:07.293781 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 16:49:07.293865 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:49:07.294031 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":469,"threads_started":1}
I20260619 16:49:07.343909 16071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:07.344208 16071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:07.348287 16071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384070a0e8b54d7f8ca0e21667a486cc: Sending DeleteTablet for 3 replicas of tablet 49172d81c08a4fc7b54dbc8d0665e537
I20260619 16:49:07.349462 16347 tablet_service.cc:1514] Processing DeleteTablet for tablet 49172d81c08a4fc7b54dbc8d0665e537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:07 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:46376
I20260619 16:49:07.349550 16212 tablet_service.cc:1514] Processing DeleteTablet for tablet 49172d81c08a4fc7b54dbc8d0665e537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:07 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:35800
I20260619 16:49:07.349789 16601 tablet_replica.cc:330] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: stopping tablet replica
I20260619 16:49:07.350000 16601 raft_consensus.cc:2229] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:07.350335 16486 tablet_service.cc:1514] Processing DeleteTablet for tablet 49172d81c08a4fc7b54dbc8d0665e537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:07 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:58688
I20260619 16:49:07.350519 16601 raft_consensus.cc:2258] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:07.350853 16599 tablet_replica.cc:330] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: stopping tablet replica
I20260619 16:49:07.351086 16599 raft_consensus.cc:2229] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:07.351270 16599 raft_consensus.cc:2258] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:07.351694 16600 tablet_replica.cc:330] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: stopping tablet replica
I20260619 16:49:07.351877 16600 raft_consensus.cc:2229] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:07.352035 16601 ts_tablet_manager.cc:1905] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:07.352113 16600 raft_consensus.cc:2258] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:07.355386 16599 ts_tablet_manager.cc:1905] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:07.355762 16600 ts_tablet_manager.cc:1905] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:07.363322 16599 ts_tablet_manager.cc:1918] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:07.363518 16599 log.cc:1196] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/49172d81c08a4fc7b54dbc8d0665e537
I20260619 16:49:07.365612 16599 ts_tablet_manager.cc:1939] T 49172d81c08a4fc7b54dbc8d0665e537 P 52f552e28f274e41b7b74b8d0e82d54d: Deleting consensus metadata
I20260619 16:49:07.368768 16601 ts_tablet_manager.cc:1918] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:07.369518 16600 ts_tablet_manager.cc:1918] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:07.369747 16600 log.cc:1196] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/49172d81c08a4fc7b54dbc8d0665e537
I20260619 16:49:07.370296 16600 ts_tablet_manager.cc:1939] T 49172d81c08a4fc7b54dbc8d0665e537 P 3d00a1be0d7643dba8db6d752f77178c: Deleting consensus metadata
I20260619 16:49:07.371469 16057 catalog_manager.cc:4841] TS 3d00a1be0d7643dba8db6d752f77178c (127.14.153.3:36271): tablet 49172d81c08a4fc7b54dbc8d0665e537 (table test-table [id=0abb465d6cbc459991361f2c1fb47510]) successfully deleted
I20260619 16:49:07.371830 16601 log.cc:1196] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/49172d81c08a4fc7b54dbc8d0665e537
I20260619 16:49:07.372421 16601 ts_tablet_manager.cc:1939] T 49172d81c08a4fc7b54dbc8d0665e537 P 2971041890164d41b88f58e811152bdc: Deleting consensus metadata
I20260619 16:49:07.373515 16056 catalog_manager.cc:4841] TS 2971041890164d41b88f58e811152bdc (127.14.153.2:33015): tablet 49172d81c08a4fc7b54dbc8d0665e537 (table test-table [id=0abb465d6cbc459991361f2c1fb47510]) successfully deleted
I20260619 16:49:07.376122 16058 catalog_manager.cc:4841] TS 52f552e28f274e41b7b74b8d0e82d54d (127.14.153.1:34037): tablet 49172d81c08a4fc7b54dbc8d0665e537 (table test-table [id=0abb465d6cbc459991361f2c1fb47510]) successfully deleted
Jun 19 16:49:07 dist-test-slave-qb36 krb5kdc[16026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:07.400453 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.368813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.14.153.1:34037)
0619 16:49:07.368993 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.368999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.378187 (+ 9188us) server_negotiation.cc:407] Connection header received
0619 16:49:07.378386 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.378391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.378487 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.378584 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:07.381223 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.382335 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.384314 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.384968 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.385073 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.391409 (+ 6336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:07.391430 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:07.391453 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:07.391507 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:07.395835 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.397505 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.397517 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.397521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.397752 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.398291 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.398296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.398300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.398843 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:07.399193 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.399808 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.400049 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 16:49:07.404610 16273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35810
I20260619 16:49:07.433843 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.416356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.14.153.62:39261)
0619 16:49:07.416600 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.416607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.416881 (+ 274us) server_negotiation.cc:407] Connection header received
0619 16:49:07.417169 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.417176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.417260 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.417337 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:07.419360 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.420490 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.422641 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.423336 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.423449 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.426868 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:07.426876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:07.426881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:07.426927 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:07.430474 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.431405 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.431412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.431417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.431560 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.432050 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.432055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.432059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.432435 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:07.432642 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.433260 (+ 618us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.433499 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260619 16:49:07.435153 16071 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50426
I20260619 16:49:07.437876 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16142
I20260619 16:49:07.473146 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.439849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.14.153.1:34037)
0619 16:49:07.440061 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.440067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.455937 (+ 15870us) server_negotiation.cc:407] Connection header received
0619 16:49:07.456945 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.456951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.457047 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.457128 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:07.459047 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.460193 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.462069 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.462760 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.462861 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.466171 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:07.466178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:07.466185 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:07.466231 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:07.469946 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.470805 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.470810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.470814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.470983 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.471445 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.471452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.471456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.471782 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:07.472009 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.472554 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.472788 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 16:49:07.474529 16273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35816
W20260619 16:49:07.475191 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16142: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:49:07.475404 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 16142
I20260619 16:49:07.501461 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16282
Jun 19 16:49:07 dist-test-slave-qb36 krb5kdc[16026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:49:07.522712 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.502281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.14.153.2:33015)
0619 16:49:07.502477 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.502484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.502788 (+ 304us) server_negotiation.cc:407] Connection header received
0619 16:49:07.502946 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.502954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.503082 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.503196 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:07.504775 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.505941 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.507859 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.508565 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.508690 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.515064 (+ 6374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:07.515081 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:07.515110 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:07.515160 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:07.519314 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.520251 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.520262 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.520266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.520434 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.520868 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.520874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.520883 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.521399 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:07.521724 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.522076 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.522324 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 16:49:07.523595 16412 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46382
W20260619 16:49:07.524329 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16282: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:49:07.524511 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 16282
I20260619 16:49:07.543550 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16421
Jun 19 16:49:07 dist-test-slave-qb36 krb5kdc[16026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:49:07.559742 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.544348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.14.153.3:36271)
0619 16:49:07.544529 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.544537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.544635 (+ 98us) server_negotiation.cc:407] Connection header received
0619 16:49:07.544868 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.544873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.544942 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.545057 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:07.546353 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.547217 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.548621 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.549028 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.549247 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.553500 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:07.553512 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:07.553528 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:07.553571 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:07.556824 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.557626 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.557633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.557635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.557770 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.558175 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.558180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.558186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.558550 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:07.558913 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.559154 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.559380 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 16:49:07.560401 16551 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58706
W20260619 16:49:07.561071 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16421: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:49:07.561254 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 16421
I20260619 16:49:07.582751 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16039
I20260619 16:49:07.583976 16111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:07.696426 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 16039
2026-06-19T16:49:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4647 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:07 dist-test-slave-qb36 krb5kdc[16641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:07 dist-test-slave-qb36 krb5kdc[16641](info): set up 2 sockets
Jun 19 16:49:07 dist-test-slave-qb36 krb5kdc[16641](info): commencing operation
krb5kdc: starting...
W20260619 16:49:08.804049 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:08 dist-test-slave-qb36 krb5kdc[16641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:49:08.987126 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:45477
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:45477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:09.309294 16654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:09.309589 16654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:09.309680 16654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:09.316967 16654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:09.317096 16654 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:49:09.317138 16654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:09.317193 16654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:09.317243 16654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:09.317305 16654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:09.327311 16654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:45477
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:45477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:09.328575 16654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:09.329833 16654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:09.341249 16660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:09.341254 16659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:09.341292 16662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:09.342883 16654 server_base.cc:1034] running on GCE node
I20260619 16:49:09.343730 16654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:09.345136 16654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:09.346374 16654 hybrid_clock.cc:648] HybridClock initialized: now 1781887749346298 us; error 41 us; skew 500 ppm
I20260619 16:49:09.346730 16654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:09 dist-test-slave-qb36 krb5kdc[16641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:09.349836 16654 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:49:09.351325 16654 webserver.cc:460] Webserver started at http://127.14.153.62:36557/ using document root <none> and password file <none>
I20260619 16:49:09.351972 16654 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:09.352053 16654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:09.352320 16654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:09.355212 16654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "e440e40df306459db37713d227ef7401"
format_stamp: "Formatted at 2026-06-19 16:49:09 on dist-test-slave-qb36"
server_key: "9d760b7c62f347f8e925c46d4882aea1"
server_key_iv: "7f2d8c4073f49c2961f6b30db9103ce4"
server_key_version: "encryptionkey@0"
I20260619 16:49:09.355943 16654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "e440e40df306459db37713d227ef7401"
format_stamp: "Formatted at 2026-06-19 16:49:09 on dist-test-slave-qb36"
server_key: "9d760b7c62f347f8e925c46d4882aea1"
server_key_iv: "7f2d8c4073f49c2961f6b30db9103ce4"
server_key_version: "encryptionkey@0"
I20260619 16:49:09.360986 16654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:09.364812 16669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:09.366137 16654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:49:09.366345 16654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "e440e40df306459db37713d227ef7401"
format_stamp: "Formatted at 2026-06-19 16:49:09 on dist-test-slave-qb36"
server_key: "9d760b7c62f347f8e925c46d4882aea1"
server_key_iv: "7f2d8c4073f49c2961f6b30db9103ce4"
server_key_version: "encryptionkey@0"
I20260619 16:49:09.366565 16654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:09.378364 16654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:09.382397 16654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:09.382680 16654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:09.403931 16654 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:45477
I20260619 16:49:09.403939 16730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:45477 every 8 connection(s)
I20260619 16:49:09.405294 16654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:09.408946 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:09.415263 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 16654
I20260619 16:49:09.415475 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:09.416290 14948 external_mini_cluster.cc:1428] Setting key b75c215648d96dd2c30fee4762a8848b
I20260619 16:49:09.420463 16731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401: Bootstrap starting.
I20260619 16:49:09.424347 16731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:09.425531 16731 log.cc:824] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:09.428836 16731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401: No bootstrap required, opened a new log
Jun 19 16:49:09 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:09.439730 16731 raft_consensus.cc:348] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e440e40df306459db37713d227ef7401" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45477 } }
I20260619 16:49:09.440199 16731 raft_consensus.cc:374] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:09.440321 16731 raft_consensus.cc:729] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e440e40df306459db37713d227ef7401, State: Initialized, Role: FOLLOWER
I20260619 16:49:09.441051 16731 consensus_queue.cc:260] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e440e40df306459db37713d227ef7401" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45477 } }
I20260619 16:49:09.441282 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:09.418531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.14.153.62:45477)
0619 16:49:09.419378 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:09.419390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:09.419434 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:49:09.421142 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:09.421166 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:09.421563 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:09.421899 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:09.423721 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.425098 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.426698 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.427454 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.427587 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:09.432247 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:09.432258 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:09.432281 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:09.432333 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:09.436405 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:09.437115 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:09.437125 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:09.437143 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:09.437389 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:09.437720 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:09.437725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:09.437729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:09.438537 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:09.438827 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:09.439324 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 16:49:09.439777 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":435,"threads_started":1}
I20260619 16:49:09.441360 16731 raft_consensus.cc:388] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:09.441501 16731 raft_consensus.cc:482] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:09.441701 16731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:09.444157 16731 raft_consensus.cc:504] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e440e40df306459db37713d227ef7401" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45477 } }
I20260619 16:49:09.444661 16731 leader_election.cc:302] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e440e40df306459db37713d227ef7401; no voters:
I20260619 16:49:09.445204 16731 leader_election.cc:288] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:09.445358 16736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:09.446019 16736 raft_consensus.cc:686] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [term 1 LEADER]: Becoming Leader. State: Replica: e440e40df306459db37713d227ef7401, State: Running, Role: LEADER
I20260619 16:49:09.446640 16736 consensus_queue.cc:237] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e440e40df306459db37713d227ef7401" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45477 } }
I20260619 16:49:09.447312 16731 sys_catalog.cc:564] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:09.449388 16738 sys_catalog.cc:455] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e440e40df306459db37713d227ef7401. Latest consensus state: current_term: 1 leader_uuid: "e440e40df306459db37713d227ef7401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e440e40df306459db37713d227ef7401" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45477 } } }
I20260619 16:49:09.449756 16738 sys_catalog.cc:458] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:09.451288 16737 sys_catalog.cc:455] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e440e40df306459db37713d227ef7401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e440e40df306459db37713d227ef7401" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45477 } } }
I20260619 16:49:09.451577 16737 sys_catalog.cc:458] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:09.452128 16741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:09.456290 16741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:09.465713 16741 catalog_manager.cc:1347] Generated new cluster ID: 52692ba1f91641ba85440f88a331263e
I20260619 16:49:09.465870 16741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:09.477552 16741 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:09.478826 16741 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:09.486733 16741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401: Generated new TSK 0
I20260619 16:49:09.487898 16741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:49:09.534603 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45477
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:09.874560 16757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:09.874840 16757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:09.874940 16757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:09.882176 16757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:09.882285 16757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:09.882372 16757 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:49:09.882426 16757 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:09.882496 16757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:09.892408 16757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.153.62:45477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:09.893724 16757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:09.894990 16757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:09.908645 16763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:09.908598 16765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:09.908681 16762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:09.909283 16757 server_base.cc:1034] running on GCE node
I20260619 16:49:09.909927 16757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:09.910944 16757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:09.912182 16757 hybrid_clock.cc:648] HybridClock initialized: now 1781887749912125 us; error 65 us; skew 500 ppm
I20260619 16:49:09.912505 16757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:09 dist-test-slave-qb36 krb5kdc[16641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:09.915657 16757 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:49:09.917078 16757 webserver.cc:460] Webserver started at http://127.14.153.1:44095/ using document root <none> and password file <none>
I20260619 16:49:09.917778 16757 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:09.917919 16757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:09.918246 16757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:09.921106 16757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49"
format_stamp: "Formatted at 2026-06-19 16:49:09 on dist-test-slave-qb36"
server_key: "e34c7fe33cdb872380b211dad1998592"
server_key_iv: "8cbdf29d5346fa6fe93776b263aae73f"
server_key_version: "encryptionkey@0"
I20260619 16:49:09.921918 16757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49"
format_stamp: "Formatted at 2026-06-19 16:49:09 on dist-test-slave-qb36"
server_key: "e34c7fe33cdb872380b211dad1998592"
server_key_iv: "8cbdf29d5346fa6fe93776b263aae73f"
server_key_version: "encryptionkey@0"
I20260619 16:49:09.927001 16757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:49:09.930389 16772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:09.931738 16757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:09.931906 16757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49"
format_stamp: "Formatted at 2026-06-19 16:49:09 on dist-test-slave-qb36"
server_key: "e34c7fe33cdb872380b211dad1998592"
server_key_iv: "8cbdf29d5346fa6fe93776b263aae73f"
server_key_version: "encryptionkey@0"
I20260619 16:49:09.932191 16757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:09.950332 16757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:09.954783 16757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:09.955111 16757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:09.957685 16757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:09.957777 16757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:09.957955 16757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:09.958088 16757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:09 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:09.981676 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:09.958561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:36971 (local address 127.14.153.62:45477)
0619 16:49:09.959143 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:09.959150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:09.960282 (+ 1132us) server_negotiation.cc:407] Connection header received
0619 16:49:09.962732 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:09.962739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:09.962824 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:09.962954 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:09.965275 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.966119 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.968710 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.969203 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.969355 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:09.974057 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:09.974062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:09.974066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:09.974119 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:09.977258 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:09.978028 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:09.978033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:09.978037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:09.978152 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:09.978809 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:09.978814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:09.978818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:09.979186 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:09.979356 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:09.980907 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0619 16:49:09.981175 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":312,"threads_started":1}
I20260619 16:49:09.982987 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:09.959238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45477 (local address 127.14.153.1:36971)
0619 16:49:09.960102 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:09.960150 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:09.962267 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:09.963258 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:09.963272 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:09.964140 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:09.965096 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:09.965117 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.966356 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.966367 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:09.968560 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:09.968572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.969501 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.969512 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:09.969954 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:09.970963 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:09.971012 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:09.973808 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:09.977388 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:09.977400 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:09.977418 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:09.977886 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:09.978308 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:09.978317 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:09.978324 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:09.978659 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:09.979348 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:09.979361 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:09.979960 (+ 599us) client_negotiation.cc:769] Sending connection context
0619 16:49:09.980362 (+ 402us) client_negotiation.cc:240] Negotiation successful
0619 16:49:09.981134 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":378,"threads_started":1}
I20260619 16:49:10.003427 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:09.988940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:58021 (local address 127.14.153.62:45477)
0619 16:49:09.989164 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:09.989174 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:09.989735 (+ 561us) server_negotiation.cc:407] Connection header received
0619 16:49:09.989898 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:09.989906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:09.989972 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:09.990091 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:09.990990 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.991748 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.994402 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.994793 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.994888 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:09.997535 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:09.997540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:09.997543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:09.997570 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.000295 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.001338 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.001345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.001350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.001521 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.001954 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.001961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.001966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.002268 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.002416 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.002795 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.003064 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:49:10.003551 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:09.988948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45477 (local address 127.14.153.1:58021)
0619 16:49:09.989633 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:09.989651 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:09.989767 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:09.990107 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:09.990112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:09.990574 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:09.990852 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:09.990860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.991911 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.991919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:09.994235 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:09.994256 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:09.995002 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:09.995011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:09.995280 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:09.995993 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:09.996023 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:09.997349 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.000435 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.000443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.000451 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.001186 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.001624 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.001630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.001634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.001776 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.002386 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.002391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.002565 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.002703 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.003056 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":355,"threads_started":1}
I20260619 16:49:10.013110 16757 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:38353
I20260619 16:49:10.013144 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:38353 every 8 connection(s)
I20260619 16:49:10.014402 16757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:10.016378 16893 heartbeater.cc:347] Connected to a master server at 127.14.153.62:45477
I20260619 16:49:10.016608 16893 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:10.017284 16893 heartbeater.cc:511] Master 127.14.153.62:45477 requested a full tablet report, sending...
I20260619 16:49:10.019177 16686 ts_manager.cc:194] Registered new tserver with Master: 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353)
I20260619 16:49:10.020582 16686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:36971
I20260619 16:49:10.023898 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 16757
I20260619 16:49:10.024049 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:10.024605 14948 external_mini_cluster.cc:1428] Setting key c96655c916f1ad09aa983bf0fbb3afb8
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:49:10.066136 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45477
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:10.397045 16897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:10.397346 16897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:10.397516 16897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:10.405097 16897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:10.405210 16897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:10.405296 16897 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:49:10.405385 16897 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:10.405541 16897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:10.415437 16897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.153.62:45477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:10.416755 16897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:10.418136 16897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:10.431587 16902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.431680 16903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.431650 16905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:10.432744 16897 server_base.cc:1034] running on GCE node
I20260619 16:49:10.433550 16897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:10.434562 16897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:10.435791 16897 hybrid_clock.cc:648] HybridClock initialized: now 1781887750435741 us; error 80 us; skew 500 ppm
I20260619 16:49:10.436170 16897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:10 dist-test-slave-qb36 krb5kdc[16641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:10.439488 16897 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:49:10.441042 16897 webserver.cc:460] Webserver started at http://127.14.153.2:39095/ using document root <none> and password file <none>
I20260619 16:49:10.441745 16897 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:10.441890 16897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:10.442220 16897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:10.445201 16897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "a1c31cda70ae4130b41a033dcee32f1f"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-qb36"
server_key: "21695a5b3f7503c54689b64597761c44"
server_key_iv: "3b531f1195b44f567146aa085119799d"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.445961 16897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "a1c31cda70ae4130b41a033dcee32f1f"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-qb36"
server_key: "21695a5b3f7503c54689b64597761c44"
server_key_iv: "3b531f1195b44f567146aa085119799d"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.451364 16897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:49:10.454931 16912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.456460 16897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:10.456663 16897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "a1c31cda70ae4130b41a033dcee32f1f"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-qb36"
server_key: "21695a5b3f7503c54689b64597761c44"
server_key_iv: "3b531f1195b44f567146aa085119799d"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.456878 16897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:10.492307 16897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:10.496627 16897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:10.496937 16897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:10.499621 16897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:10.499759 16897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.499903 16897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:10.499995 16897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:10 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:10.526314 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.500480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:58971 (local address 127.14.153.62:45477)
0619 16:49:10.500676 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.500684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.502163 (+ 1479us) server_negotiation.cc:407] Connection header received
0619 16:49:10.505253 (+ 3090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.505261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.505362 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.505459 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.508505 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.509522 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.511728 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.512530 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.512702 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.517842 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.517855 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.517859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.517892 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.521890 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.523252 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.523259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.523263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.523426 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.524073 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.524078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.524082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.524533 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.524703 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.525873 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.526054 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:49:10.528605 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.501047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45477 (local address 127.14.153.2:58971)
0619 16:49:10.501947 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.502013 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.504846 (+ 2833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.506101 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.506118 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.507156 (+ 1038us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:10.508242 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.508280 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.509744 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.509749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.511536 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.511580 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.512829 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.512838 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.513039 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.513897 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:10.513920 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:10.517580 (+ 3660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.522124 (+ 4544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.522139 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.522164 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.523080 (+ 916us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.523560 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.523568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.523571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.523942 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.524710 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.524724 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.525348 (+ 624us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.525755 (+ 407us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.526507 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":343,"threads_started":1}
I20260619 16:49:10.549247 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.534416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45477 (local address 127.14.153.2:59527)
0619 16:49:10.535315 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.535340 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.535515 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.535954 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.535963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.536665 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:10.537123 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.537140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.538230 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.538235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.539359 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.539371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.540609 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.540613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.540740 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.541383 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:10.541400 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:10.542130 (+ 730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.546353 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.546360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.546363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.546921 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.547441 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.547445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.547447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.547563 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.548412 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.548418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.548573 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.548690 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.548921 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":516,"threads_started":1}
I20260619 16:49:10.549576 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.534233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:59527 (local address 127.14.153.62:45477)
0619 16:49:10.534440 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.534447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.535441 (+ 994us) server_negotiation.cc:407] Connection header received
0619 16:49:10.535726 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.535730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.535793 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.535904 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.537278 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.538020 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.539602 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.540411 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.540556 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.542327 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.542336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.542344 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.542387 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.546186 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.547110 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.547119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.547125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.547319 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.547677 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.547685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.547691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.548274 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.548450 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.548830 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.549114 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 16:49:10.558745 16897 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:40119
I20260619 16:49:10.558776 17031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:40119 every 8 connection(s)
I20260619 16:49:10.560099 16897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:10.562304 17032 heartbeater.cc:347] Connected to a master server at 127.14.153.62:45477
I20260619 16:49:10.562537 17032 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:10.563243 17032 heartbeater.cc:511] Master 127.14.153.62:45477 requested a full tablet report, sending...
I20260619 16:49:10.564604 16686 ts_manager.cc:194] Registered new tserver with Master: a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119)
I20260619 16:49:10.565440 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 16897
I20260619 16:49:10.565605 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:10.565802 16686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:58971
I20260619 16:49:10.566131 14948 external_mini_cluster.cc:1428] Setting key 0b437071155f29ef6ca39c6fbd5c366e
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:49:10.606895 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45477
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:10.928244 17036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:10.928599 17036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:10.928778 17036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:10.936481 17036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:10.936638 17036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:10.936800 17036 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:49:10.936887 17036 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:10.937026 17036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:10.947176 17036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.153.62:45477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:10.948638 17036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:10.949951 17036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:10.963969 17041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.963888 17044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.963923 17042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:10.965024 17036 server_base.cc:1034] running on GCE node
I20260619 16:49:10.965634 17036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:10.966553 17036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:10.967780 17036 hybrid_clock.cc:648] HybridClock initialized: now 1781887750967729 us; error 44 us; skew 500 ppm
I20260619 16:49:10.968101 17036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:10 dist-test-slave-qb36 krb5kdc[16641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:10.974730 17036 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:49:10.976462 17036 webserver.cc:460] Webserver started at http://127.14.153.3:43391/ using document root <none> and password file <none>
I20260619 16:49:10.977435 17036 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:10.977588 17036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:10.977890 17036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:10.980760 17036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "ac707b8fc4c44ede8d9d1975fc1e1220"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-qb36"
server_key: "82425ab059b8559d1fc63c7ff671a0d1"
server_key_iv: "e18b2844a708ba4bffa259820a5abfbb"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.981449 17036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "ac707b8fc4c44ede8d9d1975fc1e1220"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-qb36"
server_key: "82425ab059b8559d1fc63c7ff671a0d1"
server_key_iv: "e18b2844a708ba4bffa259820a5abfbb"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.986553 17036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 16:49:10.989988 17051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.991449 17036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:10.991631 17036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "ac707b8fc4c44ede8d9d1975fc1e1220"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-qb36"
server_key: "82425ab059b8559d1fc63c7ff671a0d1"
server_key_iv: "e18b2844a708ba4bffa259820a5abfbb"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.991835 17036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:11.012624 17036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:11.017874 17036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:11.018218 17036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:11.021150 17036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:11.021320 17036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:11.021458 17036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:11.021536 17036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:11.023262 16893 heartbeater.cc:503] Master 127.14.153.62:45477 was elected leader, sending a full tablet report...
Jun 19 16:49:11 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:11.051654 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.021616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:33059 (local address 127.14.153.62:45477)
0619 16:49:11.021807 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.021816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.023642 (+ 1826us) server_negotiation.cc:407] Connection header received
0619 16:49:11.026687 (+ 3045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.026706 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.026813 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.026950 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:11.029331 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.030485 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.033987 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.034458 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.034628 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.041939 (+ 7311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:11.041950 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:11.041955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:11.042012 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:11.046575 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:11.047908 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:11.047917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:11.047924 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:11.048122 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:11.048797 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:11.048801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:11.048803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:11.049140 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:11.049308 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.050856 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.051192 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:49:11.053570 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.022336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45477 (local address 127.14.153.3:33059)
0619 16:49:11.023296 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.023400 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.025974 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.027285 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.027294 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.028094 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:11.029076 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.029097 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.030709 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.030715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.033707 (+ 2992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.033730 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.034904 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.034910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.035471 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.036732 (+ 1261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:11.036795 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:11.041564 (+ 4769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:11.046810 (+ 5246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:11.046826 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:11.046847 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:11.047715 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:11.048263 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:11.048270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:11.048279 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:11.048620 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:11.049312 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:11.049327 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:11.050076 (+ 749us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.050566 (+ 490us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.051427 (+ 861us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":374,"threads_started":1}
I20260619 16:49:11.065544 17036 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:45981
I20260619 16:49:11.065645 17170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:45981 every 8 connection(s)
I20260619 16:49:11.066931 17036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:11.069353 17171 heartbeater.cc:347] Connected to a master server at 127.14.153.62:45477
I20260619 16:49:11.069636 17171 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:11.070495 17171 heartbeater.cc:511] Master 127.14.153.62:45477 requested a full tablet report, sending...
I20260619 16:49:11.072018 16686 ts_manager.cc:194] Registered new tserver with Master: ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981)
I20260619 16:49:11.073321 16686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:33059
I20260619 16:49:11.074725 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 17036
I20260619 16:49:11.074888 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:11.075460 14948 external_mini_cluster.cc:1428] Setting key a868709a73927fb735ec1655dc5b8afb
I20260619 16:49:11.076499 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.062026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:39341 (local address 127.14.153.62:45477)
0619 16:49:11.062234 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.062241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.062734 (+ 493us) server_negotiation.cc:407] Connection header received
0619 16:49:11.062953 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.062962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.063097 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.063203 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:11.064585 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.065725 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.067114 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.067749 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.067877 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.070442 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:11.070448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:11.070453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:11.070497 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:11.073853 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:11.074523 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:11.074531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:11.074534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:11.074661 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:11.075101 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:11.075106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:11.075109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:11.075368 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:11.075530 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.075912 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.076129 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:49:11.076606 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.061359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45477 (local address 127.14.153.3:39341)
0619 16:49:11.062606 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.062630 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.062807 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.063286 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.063294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.063946 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:11.064357 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.064381 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.065910 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.065914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.066926 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.066938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.067997 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.068007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.068305 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.069178 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:11.069202 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:11.070232 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:11.073984 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:11.073990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:11.073994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:11.074397 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:11.074779 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:11.074784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:11.074788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:11.074949 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:11.075496 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:11.075501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:11.075677 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.075821 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.076149 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":807,"threads_started":1}
I20260619 16:49:11.078497 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:49:11 dist-test-slave-qb36 krb5kdc[16641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:49:11 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:11.125066 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.111779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.14.153.62:45477)
0619 16:49:11.112017 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.112025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.112247 (+ 222us) server_negotiation.cc:407] Connection header received
0619 16:49:11.112391 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.112396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.112459 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.112551 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:11.113896 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.114655 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.115930 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.116411 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.116507 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.120348 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:11.120353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:11.120356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:11.120388 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:11.122720 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:11.123369 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:11.123373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:11.123375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:11.123504 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:11.123831 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:11.123834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:11.123836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:11.124091 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:11.124281 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.124543 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.124703 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:49:11.129455 16685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:11.136451 16685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:11.160127 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.149365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.14.153.1:38353)
0619 16:49:11.150184 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.150198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.152021 (+ 1823us) server_negotiation.cc:407] Connection header received
0619 16:49:11.152125 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.152131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.152353 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.152563 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.153401 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.154871 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.157778 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.158874 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.159061 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.159276 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.159589 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.159777 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":459,"threads_started":1}
I20260619 16:49:11.160516 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.148804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:45981 (local address 127.0.0.1:44496)
0619 16:49:11.151058 (+ 2254us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.151107 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.151272 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.152088 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.152095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.152123 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.152565 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.152578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.154979 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.154984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.157533 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.157547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.159346 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.159352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.159521 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.159550 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.159602 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.159760 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1986,"mutex_wait_us":18,"thread_start_us":2654,"threads_started":1}
I20260619 16:49:11.160567 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.149050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.14.153.3:45981)
0619 16:49:11.151285 (+ 2235us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.151294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.151330 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:49:11.151548 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.151557 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.151855 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.152777 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.153111 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.154844 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.157726 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.159125 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.159711 (+ 586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.159920 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.159984 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.160206 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1514,"thread_start_us":1346,"threads_started":1}
I20260619 16:49:11.161237 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.148914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:38353 (local address 127.0.0.1:34858)
0619 16:49:11.151768 (+ 2854us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.151785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.151901 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.152770 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.152774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.152798 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.153200 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.153215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.155079 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.155087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.157585 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.157607 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.159105 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.159112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.159326 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.159387 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.159521 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.160886 (+ 1365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2726,"mutex_wait_us":106,"thread_start_us":1772,"threads_started":1}
I20260619 16:49:11.162307 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.151640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:40119 (local address 127.0.0.1:51480)
0619 16:49:11.153605 (+ 1965us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.153624 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.153756 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.154222 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.154230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.154252 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.154574 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.154582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.156337 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.156343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.158526 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.158539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.161553 (+ 3014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.161564 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.161687 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.161708 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.161815 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.161940 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1856,"mutex_wait_us":8,"thread_start_us":1733,"threads_started":1}
I20260619 16:49:11.162845 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.152105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.14.153.2:40119)
0619 16:49:11.153170 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.153182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.153734 (+ 552us) server_negotiation.cc:407] Connection header received
0619 16:49:11.153899 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.153904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.154047 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.154230 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.154744 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.156162 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.158692 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.161002 (+ 2310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.161148 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.161338 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.161865 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.162026 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":793,"threads_started":1}
I20260619 16:49:11.164646 17101 tablet_service.cc:1467] Processing CreateTablet for tablet 2631eea138354239b6585ae84d6bf068 (DEFAULT_TABLE table=test-table [id=7873bfe415824117bb120eb8a3881955]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:11.164661 16823 tablet_service.cc:1467] Processing CreateTablet for tablet 2631eea138354239b6585ae84d6bf068 (DEFAULT_TABLE table=test-table [id=7873bfe415824117bb120eb8a3881955]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:11.166038 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2631eea138354239b6585ae84d6bf068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.166074 17101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2631eea138354239b6585ae84d6bf068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.166039 16962 tablet_service.cc:1467] Processing CreateTablet for tablet 2631eea138354239b6585ae84d6bf068 (DEFAULT_TABLE table=test-table [id=7873bfe415824117bb120eb8a3881955]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:11.167337 16962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2631eea138354239b6585ae84d6bf068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.177721 17190 tablet_bootstrap.cc:492] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Bootstrap starting.
I20260619 16:49:11.180594 17191 tablet_bootstrap.cc:492] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: Bootstrap starting.
I20260619 16:49:11.182642 17190 tablet_bootstrap.cc:654] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.183555 17192 tablet_bootstrap.cc:492] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: Bootstrap starting.
I20260619 16:49:11.184001 17190 log.cc:824] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:11.184197 17191 tablet_bootstrap.cc:654] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.185324 17191 log.cc:824] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:11.187343 17190 tablet_bootstrap.cc:492] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: No bootstrap required, opened a new log
I20260619 16:49:11.187376 17192 tablet_bootstrap.cc:654] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.187616 17190 ts_tablet_manager.cc:1397] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 16:49:11.188544 17192 log.cc:824] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:11.188802 17191 tablet_bootstrap.cc:492] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: No bootstrap required, opened a new log
I20260619 16:49:11.189064 17191 ts_tablet_manager.cc:1397] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 16:49:11.190944 17192 tablet_bootstrap.cc:492] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: No bootstrap required, opened a new log
I20260619 16:49:11.191221 17192 ts_tablet_manager.cc:1397] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260619 16:49:11.198262 17192 raft_consensus.cc:348] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } }
I20260619 16:49:11.198734 17192 raft_consensus.cc:374] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.198482 17190 raft_consensus.cc:348] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } }
I20260619 16:49:11.198835 17192 raft_consensus.cc:729] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c31cda70ae4130b41a033dcee32f1f, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.198910 17190 raft_consensus.cc:374] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.199002 17190 raft_consensus.cc:729] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6fbfbc814547f5a8e928ee2ebdbf49, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.199709 17192 consensus_queue.cc:260] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } }
I20260619 16:49:11.199672 17190 consensus_queue.cc:260] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } }
I20260619 16:49:11.200157 17191 raft_consensus.cc:348] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } }
I20260619 16:49:11.200928 17032 heartbeater.cc:503] Master 127.14.153.62:45477 was elected leader, sending a full tablet report...
I20260619 16:49:11.201493 17192 ts_tablet_manager.cc:1428] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 16:49:11.202282 17191 raft_consensus.cc:374] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.202404 17191 raft_consensus.cc:729] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac707b8fc4c44ede8d9d1975fc1e1220, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.203079 17190 ts_tablet_manager.cc:1428] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 16:49:11.203155 17191 consensus_queue.cc:260] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } }
I20260619 16:49:11.204336 17171 heartbeater.cc:503] Master 127.14.153.62:45477 was elected leader, sending a full tablet report...
I20260619 16:49:11.204982 17191 ts_tablet_manager.cc:1428] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260619 16:49:11.266999 16894 tablet.cc:2369] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:11.312352 17033 tablet.cc:2369] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:11.319532 17172 tablet.cc:2369] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:11.371517 17197 raft_consensus.cc:482] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:11.371868 17197 raft_consensus.cc:504] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } }
I20260619 16:49:11.373935 17197 leader_election.cc:288] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981), a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119)
I20260619 16:49:11.380811 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.374276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:52237 (local address 127.14.153.3:45981)
0619 16:49:11.374518 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.374528 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.375424 (+ 896us) server_negotiation.cc:407] Connection header received
0619 16:49:11.375646 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.375650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.375715 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.375808 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.376410 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.377244 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.379076 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.379859 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.380052 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.380126 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.380403 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.380533 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:49:11.380818 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.374690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:34765 (local address 127.14.153.2:40119)
0619 16:49:11.374939 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.374948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.375544 (+ 596us) server_negotiation.cc:407] Connection header received
0619 16:49:11.375747 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.375752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.375816 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.375909 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.376491 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.377309 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.379114 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.379861 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.380048 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.380131 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.380425 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.380554 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:49:11.380990 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.374410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:40119 (local address 127.14.153.1:34765)
0619 16:49:11.375437 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.375451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.375609 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.375939 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.375943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.375991 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.376334 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.376342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.377446 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.377451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.378940 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.378951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.380126 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.380131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.380275 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.380296 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.380402 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.380519 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":26,"thread_start_us":765,"threads_started":1}
I20260619 16:49:11.381305 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.374150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:45981 (local address 127.14.153.1:52237)
0619 16:49:11.375306 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.375322 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.375490 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.375833 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.375837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.375866 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.376267 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.376276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.377365 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.377372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.378906 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.378919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.380081 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.380087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.380252 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.380277 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.380380 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.380502 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":20,"thread_start_us":776,"threads_started":1}
I20260619 16:49:11.382061 16982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2631eea138354239b6585ae84d6bf068" candidate_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c31cda70ae4130b41a033dcee32f1f" is_pre_election: true
I20260619 16:49:11.382196 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2631eea138354239b6585ae84d6bf068" candidate_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" is_pre_election: true
I20260619 16:49:11.382670 17121 raft_consensus.cc:2454] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b6fbfbc814547f5a8e928ee2ebdbf49 in term 0.
I20260619 16:49:11.382668 16982 raft_consensus.cc:2454] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b6fbfbc814547f5a8e928ee2ebdbf49 in term 0.
I20260619 16:49:11.383337 16776 leader_election.cc:302] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6fbfbc814547f5a8e928ee2ebdbf49, a1c31cda70ae4130b41a033dcee32f1f; no voters:
I20260619 16:49:11.383666 17197 raft_consensus.cc:2781] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:11.383828 17197 raft_consensus.cc:482] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:11.383939 17197 raft_consensus.cc:3037] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.386488 17197 raft_consensus.cc:504] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } }
I20260619 16:49:11.387202 17197 leader_election.cc:288] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [CANDIDATE]: Term 1 election: Requested vote from peers ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981), a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119)
I20260619 16:49:11.387553 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2631eea138354239b6585ae84d6bf068" candidate_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220"
I20260619 16:49:11.387634 16982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2631eea138354239b6585ae84d6bf068" candidate_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c31cda70ae4130b41a033dcee32f1f"
I20260619 16:49:11.387768 17121 raft_consensus.cc:3037] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.387876 16982 raft_consensus.cc:3037] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.389982 16982 raft_consensus.cc:2454] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b6fbfbc814547f5a8e928ee2ebdbf49 in term 1.
I20260619 16:49:11.389978 17121 raft_consensus.cc:2454] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b6fbfbc814547f5a8e928ee2ebdbf49 in term 1.
I20260619 16:49:11.390379 16776 leader_election.cc:302] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6fbfbc814547f5a8e928ee2ebdbf49, a1c31cda70ae4130b41a033dcee32f1f; no voters:
I20260619 16:49:11.390664 17197 raft_consensus.cc:2781] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:11.391134 17197 raft_consensus.cc:686] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 LEADER]: Becoming Leader. State: Replica: 5b6fbfbc814547f5a8e928ee2ebdbf49, State: Running, Role: LEADER
I20260619 16:49:11.391630 17197 consensus_queue.cc:237] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } }
I20260619 16:49:11.396003 16686 catalog_manager.cc:5495] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1). New cstate: current_term: 1 leader_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:49:11 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887749, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:11.482640 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.469624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45477 (local address 127.0.0.1:48130)
0619 16:49:11.470210 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.470227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.470359 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.470716 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.470719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.471551 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:11.471822 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.471830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.472827 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.472831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.473784 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.473794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.474490 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.474494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.474618 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.475302 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:11.475322 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:11.477879 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:11.480464 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:11.480474 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:11.480481 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:11.480890 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:11.481208 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:11.481212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:11.481214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:11.481350 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:11.481815 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:11.481822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:11.481966 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.482093 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.482302 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":325,"threads_started":1}
I20260619 16:49:11.482678 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.469491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.14.153.62:45477)
0619 16:49:11.469746 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.469754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.470340 (+ 586us) server_negotiation.cc:407] Connection header received
0619 16:49:11.470450 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.470453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.470521 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.470612 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:11.471964 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.472712 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.473903 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.474324 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.474415 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.478049 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:11.478054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:11.478057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:11.478084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:11.480342 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:11.480990 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:11.480994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:11.480996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:11.481102 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:11.481438 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:11.481441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:11.481443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:11.481704 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:11.481805 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.482175 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.482419 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":87}
I20260619 16:49:11.487823 16686 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:48130:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:49:11.488837 16686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:11.495839 16823 tablet_service.cc:1467] Processing CreateTablet for tablet f8a9ee3c5a054fda895517c0717953e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf59069cca114850b534a4111fad634c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:11.496902 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a9ee3c5a054fda895517c0717953e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.496660 16962 tablet_service.cc:1467] Processing CreateTablet for tablet f8a9ee3c5a054fda895517c0717953e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf59069cca114850b534a4111fad634c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:11.496682 17101 tablet_service.cc:1467] Processing CreateTablet for tablet f8a9ee3c5a054fda895517c0717953e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf59069cca114850b534a4111fad634c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:11.497306 16962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a9ee3c5a054fda895517c0717953e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.497491 17101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a9ee3c5a054fda895517c0717953e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.502242 17192 tablet_bootstrap.cc:492] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: Bootstrap starting.
I20260619 16:49:11.503641 17192 tablet_bootstrap.cc:654] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.504390 17191 tablet_bootstrap.cc:492] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: Bootstrap starting.
I20260619 16:49:11.505990 17191 tablet_bootstrap.cc:654] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.506310 17192 tablet_bootstrap.cc:492] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: No bootstrap required, opened a new log
I20260619 16:49:11.506433 17192 ts_tablet_manager.cc:1397] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:49:11.506981 17190 tablet_bootstrap.cc:492] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Bootstrap starting.
I20260619 16:49:11.507223 17192 raft_consensus.cc:348] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.507449 17192 raft_consensus.cc:374] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.507542 17192 raft_consensus.cc:729] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c31cda70ae4130b41a033dcee32f1f, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.507773 17192 consensus_queue.cc:260] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.508306 17196 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.508649 17190 tablet_bootstrap.cc:654] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.508706 17196 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.508991 17192 ts_tablet_manager.cc:1428] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 16:49:11.511013 17190 tablet_bootstrap.cc:492] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: No bootstrap required, opened a new log
I20260619 16:49:11.511015 17191 tablet_bootstrap.cc:492] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: No bootstrap required, opened a new log
I20260619 16:49:11.511173 17190 ts_tablet_manager.cc:1397] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 16:49:11.511173 17191 ts_tablet_manager.cc:1397] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260619 16:49:11.511926 17190 raft_consensus.cc:348] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.511914 17191 raft_consensus.cc:348] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.512141 17190 raft_consensus.cc:374] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.512156 17191 raft_consensus.cc:374] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.512230 17190 raft_consensus.cc:729] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6fbfbc814547f5a8e928ee2ebdbf49, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.512255 17191 raft_consensus.cc:729] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac707b8fc4c44ede8d9d1975fc1e1220, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.512459 17191 consensus_queue.cc:260] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.512459 17190 consensus_queue.cc:260] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.512996 17191 ts_tablet_manager.cc:1428] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 16:49:11.513058 17190 ts_tablet_manager.cc:1428] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 16:49:11.512907 17197 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.513198 17197 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.513506 17198 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.513935 17198 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.568972 17198 raft_consensus.cc:482] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:11.569229 17198 raft_consensus.cc:504] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.571147 17198 leader_election.cc:288] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353), a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119)
I20260619 16:49:11.578006 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.571491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:40583 (local address 127.14.153.1:38353)
0619 16:49:11.571691 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.571695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.572135 (+ 440us) server_negotiation.cc:407] Connection header received
0619 16:49:11.572346 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.572351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.572431 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.572535 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.573223 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.574169 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.576149 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.577218 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.577344 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.577432 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.577728 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.577822 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:49:11.578145 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.571428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:38353 (local address 127.14.153.3:40583)
0619 16:49:11.571965 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.571986 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.572169 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.572676 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.572680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.572712 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.573024 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.573035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.574310 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.574316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.575974 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.575985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.577372 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.577379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.577547 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.577586 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.577686 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.577820 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":309,"threads_started":1}
I20260619 16:49:11.579210 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.572294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:39147 (local address 127.14.153.2:40119)
0619 16:49:11.572469 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.572477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.572805 (+ 328us) server_negotiation.cc:407] Connection header received
0619 16:49:11.572938 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.572946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.573031 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.573139 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.573794 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.574780 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.576845 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.577971 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.578156 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.578239 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.578786 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.578909 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:49:11.579047 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a9ee3c5a054fda895517c0717953e7" candidate_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" is_pre_election: true
I20260619 16:49:11.579265 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.572149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:40119 (local address 127.14.153.3:39147)
0619 16:49:11.572704 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.572718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.572823 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.573192 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.573197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.573221 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.573622 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.573633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.574930 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.574934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.576668 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.576681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.578511 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.578515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.578633 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.578654 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.578759 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.578879 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":1,"thread_start_us":286,"threads_started":1}
I20260619 16:49:11.579494 16848 raft_consensus.cc:2454] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac707b8fc4c44ede8d9d1975fc1e1220 in term 0.
I20260619 16:49:11.579957 16982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a9ee3c5a054fda895517c0717953e7" candidate_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c31cda70ae4130b41a033dcee32f1f" is_pre_election: true
I20260619 16:49:11.580080 17055 leader_election.cc:302] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6fbfbc814547f5a8e928ee2ebdbf49, ac707b8fc4c44ede8d9d1975fc1e1220; no voters:
I20260619 16:49:11.580201 16982 raft_consensus.cc:2454] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac707b8fc4c44ede8d9d1975fc1e1220 in term 0.
I20260619 16:49:11.580737 17198 raft_consensus.cc:2781] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:11.580833 17198 raft_consensus.cc:482] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:11.580891 17198 raft_consensus.cc:3037] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.582811 17198 raft_consensus.cc:504] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.583345 17198 leader_election.cc:288] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [CANDIDATE]: Term 1 election: Requested vote from peers 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353), a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119)
I20260619 16:49:11.583828 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a9ee3c5a054fda895517c0717953e7" candidate_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49"
I20260619 16:49:11.583935 16982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a9ee3c5a054fda895517c0717953e7" candidate_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c31cda70ae4130b41a033dcee32f1f"
I20260619 16:49:11.584031 16848 raft_consensus.cc:3037] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.584203 16982 raft_consensus.cc:3037] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.586526 16982 raft_consensus.cc:2454] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac707b8fc4c44ede8d9d1975fc1e1220 in term 1.
I20260619 16:49:11.586627 16848 raft_consensus.cc:2454] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac707b8fc4c44ede8d9d1975fc1e1220 in term 1.
I20260619 16:49:11.586933 17055 leader_election.cc:302] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1c31cda70ae4130b41a033dcee32f1f, ac707b8fc4c44ede8d9d1975fc1e1220; no voters:
I20260619 16:49:11.587221 17198 raft_consensus.cc:2781] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:11.587623 17198 raft_consensus.cc:686] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 LEADER]: Becoming Leader. State: Replica: ac707b8fc4c44ede8d9d1975fc1e1220, State: Running, Role: LEADER
I20260619 16:49:11.588092 17198 consensus_queue.cc:237] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.591190 17214 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: TxnStatusTablet state changed. Reason: New leader ac707b8fc4c44ede8d9d1975fc1e1220. Latest consensus state: current_term: 1 leader_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.591463 17214 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:11.592178 17216 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:49:11.592450 17216 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:49:11.592597 16686 catalog_manager.cc:5495] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3). New cstate: current_term: 1 leader_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } health_report { overall_health: HEALTHY } } }
I20260619 16:49:11.619310 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.612958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:45981 (local address 127.0.0.1:47408)
0619 16:49:11.613191 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.613208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.613310 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.613801 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.613804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.613813 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:11.613960 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.613966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.615494 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.615498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.616521 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.616534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.617495 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.617500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.617673 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.617727 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.618804 (+ 1077us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.618821 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.618892 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.619005 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:49:11.619369 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.613075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.14.153.3:45981)
0619 16:49:11.613268 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.613274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.613296 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:49:11.613500 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.613506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.613632 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.613793 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.614107 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.615364 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.616668 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.617335 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.617458 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.617909 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.618676 (+ 767us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.618829 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.618936 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.619018 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:49:11.627220 16982 raft_consensus.cc:1261] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Refusing update from remote peer ac707b8fc4c44ede8d9d1975fc1e1220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.627221 16848 raft_consensus.cc:1261] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Refusing update from remote peer ac707b8fc4c44ede8d9d1975fc1e1220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.628226 17198 consensus_queue.cc:1035] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.628634 17214 consensus_queue.cc:1035] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.645160 17196 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: TxnStatusTablet state changed. Reason: New leader ac707b8fc4c44ede8d9d1975fc1e1220. Latest consensus state: current_term: 1 leader_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.645109 17197 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: TxnStatusTablet state changed. Reason: New leader ac707b8fc4c44ede8d9d1975fc1e1220. Latest consensus state: current_term: 1 leader_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.645511 17196 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.645591 17197 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.648095 17219 mvcc.cc:205] Tried to move back new op lower bound from 7298612230655008768 to 7298612230508417024. Current Snapshot: MvccSnapshot[applied={T|T < 7298612230655008768}]
I20260619 16:49:11.649253 17198 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.649541 17198 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:11.652062 17220 mvcc.cc:205] Tried to move back new op lower bound from 7298612230655008768 to 7298612230508417024. Current Snapshot: MvccSnapshot[applied={T|T < 7298612230655008768}]
I20260619 16:49:11.660434 17197 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.660748 17197 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P 5b6fbfbc814547f5a8e928ee2ebdbf49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.663226 17214 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.663594 17214 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P ac707b8fc4c44ede8d9d1975fc1e1220: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:11.669483 17196 tablet_replica.cc:439] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } } }
I20260619 16:49:11.669785 17196 tablet_replica.cc:442] T f8a9ee3c5a054fda895517c0717953e7 P a1c31cda70ae4130b41a033dcee32f1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.677886 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.668665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.14.153.1:38353)
0619 16:49:11.668855 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.668862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.671872 (+ 3010us) server_negotiation.cc:407] Connection header received
0619 16:49:11.672078 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.672081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.672157 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.672286 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.672772 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.673592 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.675649 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.676040 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.676135 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.676653 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.677201 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.677313 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.677438 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.677508 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:49:11.695142 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.689151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:38215 (local address 127.14.153.3:45981)
0619 16:49:11.689396 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.689404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.689726 (+ 322us) server_negotiation.cc:407] Connection header received
0619 16:49:11.689993 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.689997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.690074 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.690184 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.690730 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.691638 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.693119 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.693631 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.693796 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.694289 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.694393 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.694502 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.694660 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.694744 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:49:11.695245 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.689011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:45981 (local address 127.14.153.1:38215)
0619 16:49:11.689612 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.689629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.689820 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.690214 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.690220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.690232 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:11.690565 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.690574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.691780 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.691789 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.692963 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.692975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.693829 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.693836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.694003 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.694060 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.694503 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.694517 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.694631 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.694773 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":295,"threads_started":1}
I20260619 16:49:11.705874 17121 raft_consensus.cc:1261] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Refusing update from remote peer 5b6fbfbc814547f5a8e928ee2ebdbf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.706547 16982 raft_consensus.cc:1261] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Refusing update from remote peer 5b6fbfbc814547f5a8e928ee2ebdbf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.706678 17201 consensus_queue.cc:1035] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.707207 17197 consensus_queue.cc:1035] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.743021 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.736559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:38353 (local address 127.14.153.3:41973)
0619 16:49:11.737279 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.737296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.737417 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.737743 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.737748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.737759 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:11.738249 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.738264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.739565 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.739573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.740761 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.740773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.741527 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.741536 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.741671 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.741726 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.742269 (+ 543us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.742291 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.742401 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.742560 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":401,"threads_started":1}
I20260619 16:49:11.743011 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.737016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:41973 (local address 127.14.153.1:38353)
0619 16:49:11.737220 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.737227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.737376 (+ 149us) server_negotiation.cc:407] Connection header received
0619 16:49:11.737564 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.737569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.737639 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.737728 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.738461 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.739369 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.740909 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.741335 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.741481 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.741896 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.742037 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.742130 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.742656 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.742734 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:49:11.796453 16686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:11.796773 16686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:11.800741 16686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401: Sending DeleteTablet for 3 replicas of tablet 2631eea138354239b6585ae84d6bf068
I20260619 16:49:11.801875 16823 tablet_service.cc:1514] Processing DeleteTablet for tablet 2631eea138354239b6585ae84d6bf068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:11 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:34858
I20260619 16:49:11.801882 17101 tablet_service.cc:1514] Processing DeleteTablet for tablet 2631eea138354239b6585ae84d6bf068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:11 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260619 16:49:11.803740 16962 tablet_service.cc:1514] Processing DeleteTablet for tablet 2631eea138354239b6585ae84d6bf068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:11 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:51480
I20260619 16:49:11.811935 17242 tablet_replica.cc:330] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: stopping tablet replica
I20260619 16:49:11.812439 17242 raft_consensus.cc:2229] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:11.812834 17242 raft_consensus.cc:2258] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:11.813211 16686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:11.813933 16686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:11.815603 17242 ts_tablet_manager.cc:1905] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:11.827248 16961 tablet_service.cc:1467] Processing CreateTablet for tablet f393a78776294ff4bb73773973814886 (DEFAULT_TABLE table=test-table [id=0f10832a4d334814b8d13dce44c96ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:11.827325 17101 tablet_service.cc:1467] Processing CreateTablet for tablet f393a78776294ff4bb73773973814886 (DEFAULT_TABLE table=test-table [id=0f10832a4d334814b8d13dce44c96ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:11.827780 17101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f393a78776294ff4bb73773973814886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.827826 16961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f393a78776294ff4bb73773973814886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.831432 16822 tablet_service.cc:1467] Processing CreateTablet for tablet f393a78776294ff4bb73773973814886 (DEFAULT_TABLE table=test-table [id=0f10832a4d334814b8d13dce44c96ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:11.832015 16822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f393a78776294ff4bb73773973814886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.835111 17191 tablet_bootstrap.cc:492] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: Bootstrap starting.
I20260619 16:49:11.837033 17191 tablet_bootstrap.cc:654] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.839995 17242 ts_tablet_manager.cc:1918] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:11.840158 17242 log.cc:1196] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/2631eea138354239b6585ae84d6bf068
I20260619 16:49:11.840709 17242 ts_tablet_manager.cc:1939] T 2631eea138354239b6585ae84d6bf068 P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting consensus metadata
I20260619 16:49:11.842556 16671 catalog_manager.cc:4841] TS ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981): tablet 2631eea138354239b6585ae84d6bf068 (table test-table [id=7873bfe415824117bb120eb8a3881955]) successfully deleted
I20260619 16:49:11.843156 17192 tablet_bootstrap.cc:492] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: Bootstrap starting.
I20260619 16:49:11.844342 17191 tablet_bootstrap.cc:492] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: No bootstrap required, opened a new log
I20260619 16:49:11.844514 17191 ts_tablet_manager.cc:1397] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 16:49:11.844800 17192 tablet_bootstrap.cc:654] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.845366 17191 raft_consensus.cc:348] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.845621 17191 raft_consensus.cc:374] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.845722 17191 raft_consensus.cc:729] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac707b8fc4c44ede8d9d1975fc1e1220, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.846028 17191 consensus_queue.cc:260] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.846738 17191 ts_tablet_manager.cc:1428] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:11.851989 17190 tablet_bootstrap.cc:492] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Bootstrap starting.
I20260619 16:49:11.853231 17192 tablet_bootstrap.cc:492] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: No bootstrap required, opened a new log
I20260619 16:49:11.853474 17192 ts_tablet_manager.cc:1397] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 16:49:11.853921 17190 tablet_bootstrap.cc:654] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.854467 17192 raft_consensus.cc:348] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.854804 17192 raft_consensus.cc:374] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.854941 17192 raft_consensus.cc:729] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c31cda70ae4130b41a033dcee32f1f, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.855273 17192 consensus_queue.cc:260] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.855948 17192 ts_tablet_manager.cc:1428] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:11.872656 17243 tablet_replica.cc:330] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: stopping tablet replica
I20260619 16:49:11.872898 17190 tablet_bootstrap.cc:492] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: No bootstrap required, opened a new log
I20260619 16:49:11.873088 17190 ts_tablet_manager.cc:1397] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.005s
I20260619 16:49:11.873136 17243 raft_consensus.cc:2229] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:11.873528 17243 raft_consensus.cc:2258] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:11.874274 17241 tablet_replica.cc:330] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: stopping tablet replica
I20260619 16:49:11.874688 17241 raft_consensus.cc:2229] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:11.875365 17241 raft_consensus.cc:2258] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:11.875938 17190 raft_consensus.cc:348] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.876191 17190 raft_consensus.cc:374] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.876281 17190 raft_consensus.cc:729] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6fbfbc814547f5a8e928ee2ebdbf49, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.876542 17243 ts_tablet_manager.cc:1905] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:11.876559 17190 consensus_queue.cc:260] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:11.879518 17190 ts_tablet_manager.cc:1428] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260619 16:49:11.880282 17241 ts_tablet_manager.cc:1905] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:11.888429 17241 ts_tablet_manager.cc:1918] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:11.888616 17241 log.cc:1196] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/2631eea138354239b6585ae84d6bf068
I20260619 16:49:11.889161 17241 ts_tablet_manager.cc:1939] T 2631eea138354239b6585ae84d6bf068 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting consensus metadata
I20260619 16:49:11.890913 17243 ts_tablet_manager.cc:1918] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:11.890895 16673 catalog_manager.cc:4841] TS 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353): tablet 2631eea138354239b6585ae84d6bf068 (table test-table [id=7873bfe415824117bb120eb8a3881955]) successfully deleted
I20260619 16:49:11.891121 17243 log.cc:1196] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/2631eea138354239b6585ae84d6bf068
I20260619 16:49:11.891647 17243 ts_tablet_manager.cc:1939] T 2631eea138354239b6585ae84d6bf068 P a1c31cda70ae4130b41a033dcee32f1f: Deleting consensus metadata
I20260619 16:49:11.893256 16673 catalog_manager.cc:4841] TS a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119): tablet 2631eea138354239b6585ae84d6bf068 (table test-table [id=7873bfe415824117bb120eb8a3881955]) successfully deleted
I20260619 16:49:12.068934 17225 raft_consensus.cc:482] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:12.069173 17225 raft_consensus.cc:504] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:12.070106 17225 leader_election.cc:288] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353), a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119)
I20260619 16:49:12.071172 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f393a78776294ff4bb73773973814886" candidate_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" is_pre_election: true
I20260619 16:49:12.071465 16847 raft_consensus.cc:2454] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac707b8fc4c44ede8d9d1975fc1e1220 in term 0.
I20260619 16:49:12.072103 16982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f393a78776294ff4bb73773973814886" candidate_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c31cda70ae4130b41a033dcee32f1f" is_pre_election: true
I20260619 16:49:12.072346 16982 raft_consensus.cc:2454] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac707b8fc4c44ede8d9d1975fc1e1220 in term 0.
I20260619 16:49:12.073448 17055 leader_election.cc:302] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1c31cda70ae4130b41a033dcee32f1f, ac707b8fc4c44ede8d9d1975fc1e1220; no voters:
I20260619 16:49:12.073961 17225 raft_consensus.cc:2781] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:12.074079 17225 raft_consensus.cc:482] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:12.074173 17225 raft_consensus.cc:3037] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:12.077083 17225 raft_consensus.cc:504] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:12.078305 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f393a78776294ff4bb73773973814886" candidate_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49"
I20260619 16:49:12.078809 16847 raft_consensus.cc:3037] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:12.082122 16847 raft_consensus.cc:2454] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac707b8fc4c44ede8d9d1975fc1e1220 in term 1.
I20260619 16:49:12.082751 17055 leader_election.cc:302] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6fbfbc814547f5a8e928ee2ebdbf49, ac707b8fc4c44ede8d9d1975fc1e1220; no voters:
I20260619 16:49:12.083109 17198 raft_consensus.cc:2781] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:12.083263 17198 raft_consensus.cc:686] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 LEADER]: Becoming Leader. State: Replica: ac707b8fc4c44ede8d9d1975fc1e1220, State: Running, Role: LEADER
I20260619 16:49:12.083537 17198 consensus_queue.cc:237] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:12.086352 16982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f393a78776294ff4bb73773973814886" candidate_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c31cda70ae4130b41a033dcee32f1f"
I20260619 16:49:12.086592 16982 raft_consensus.cc:3037] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:12.087018 16686 catalog_manager.cc:5495] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3). New cstate: current_term: 1 leader_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } health_report { overall_health: HEALTHY } } }
I20260619 16:49:12.090018 16982 raft_consensus.cc:2454] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac707b8fc4c44ede8d9d1975fc1e1220 in term 1.
I20260619 16:49:12.091320 17225 leader_election.cc:288] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [CANDIDATE]: Term 1 election: Requested vote from peers 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353), a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119)
I20260619 16:49:12.114441 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:12.108458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.14.153.3:45981)
0619 16:49:12.108604 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:12.108609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:12.108714 (+ 105us) server_negotiation.cc:407] Connection header received
0619 16:49:12.108907 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:12.108910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:12.108973 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:12.109046 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:12.109446 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.110332 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.112722 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.113143 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.113235 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:12.113724 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:12.113842 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:12.113907 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:12.114084 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:49:12.114149 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:49:12.127980 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:12.122540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:43195 (local address 127.14.153.3:45981)
0619 16:49:12.122693 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:12.122697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:12.122772 (+ 75us) server_negotiation.cc:407] Connection header received
0619 16:49:12.122934 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:12.122937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:12.122995 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:12.123117 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:12.123723 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.124413 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.126355 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.126750 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.126847 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:12.127316 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:12.127415 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:12.127485 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:12.127641 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 16:49:12.127712 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:49:12.128192 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:12.122476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:45981 (local address 127.14.153.3:43195)
0619 16:49:12.122673 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:12.122702 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:12.122828 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:12.123407 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:12.123411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:12.123426 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:12.123599 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:12.123609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.124555 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.124562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:12.126139 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:12.126155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.126906 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.126913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:12.127114 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:12.127165 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:12.127519 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:12.127531 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 16:49:12.127618 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:49:12.127781 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:49:12.137557 16982 raft_consensus.cc:1261] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Refusing update from remote peer ac707b8fc4c44ede8d9d1975fc1e1220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:12.137693 16847 raft_consensus.cc:1261] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Refusing update from remote peer ac707b8fc4c44ede8d9d1975fc1e1220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:12.138211 17225 consensus_queue.cc:1035] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:12.138510 17198 consensus_queue.cc:1035] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:49:12.162511 17219 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:49:13.762831 16686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:13.763101 16686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:13.766907 16686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401: Sending DeleteTablet for 3 replicas of tablet f393a78776294ff4bb73773973814886
I20260619 16:49:13.767701 16823 tablet_service.cc:1514] Processing DeleteTablet for tablet f393a78776294ff4bb73773973814886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:13 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:34858
I20260619 16:49:13.767751 17101 tablet_service.cc:1514] Processing DeleteTablet for tablet f393a78776294ff4bb73773973814886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:13 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260619 16:49:13.767812 16962 tablet_service.cc:1514] Processing DeleteTablet for tablet f393a78776294ff4bb73773973814886 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:13 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:51480
I20260619 16:49:13.780596 17270 tablet_replica.cc:330] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: stopping tablet replica
I20260619 16:49:13.780625 17272 tablet_replica.cc:330] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: stopping tablet replica
I20260619 16:49:13.780844 17272 raft_consensus.cc:2229] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:13.781087 17270 raft_consensus.cc:2229] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:13.781148 17272 raft_consensus.cc:2258] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:13.781706 17270 raft_consensus.cc:2258] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:13.781770 17271 tablet_replica.cc:330] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: stopping tablet replica
I20260619 16:49:13.781970 17271 raft_consensus.cc:2229] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:13.782611 17271 raft_consensus.cc:2258] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:13.782907 17270 ts_tablet_manager.cc:1905] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:13.783653 17271 ts_tablet_manager.cc:1905] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:13.786926 17272 ts_tablet_manager.cc:1905] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:13.790377 17271 ts_tablet_manager.cc:1918] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:13.790526 17271 log.cc:1196] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/f393a78776294ff4bb73773973814886
I20260619 16:49:13.790632 17270 ts_tablet_manager.cc:1918] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:13.790771 17270 log.cc:1196] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/f393a78776294ff4bb73773973814886
I20260619 16:49:13.791002 17271 ts_tablet_manager.cc:1939] T f393a78776294ff4bb73773973814886 P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting consensus metadata
I20260619 16:49:13.791236 17270 ts_tablet_manager.cc:1939] T f393a78776294ff4bb73773973814886 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting consensus metadata
I20260619 16:49:13.792239 16671 catalog_manager.cc:4841] TS ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981): tablet f393a78776294ff4bb73773973814886 (table test-table [id=0f10832a4d334814b8d13dce44c96ad3]) successfully deleted
I20260619 16:49:13.792944 16673 catalog_manager.cc:4841] TS 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353): tablet f393a78776294ff4bb73773973814886 (table test-table [id=0f10832a4d334814b8d13dce44c96ad3]) successfully deleted
I20260619 16:49:13.796502 17272 ts_tablet_manager.cc:1918] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:13.796691 17272 log.cc:1196] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/f393a78776294ff4bb73773973814886
I20260619 16:49:13.797092 17272 ts_tablet_manager.cc:1939] T f393a78776294ff4bb73773973814886 P a1c31cda70ae4130b41a033dcee32f1f: Deleting consensus metadata
I20260619 16:49:13.798015 16673 catalog_manager.cc:4841] TS a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119): tablet f393a78776294ff4bb73773973814886 (table test-table [id=0f10832a4d334814b8d13dce44c96ad3]) successfully deleted
Jun 19 16:49:13 dist-test-slave-qb36 krb5kdc[16641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:49:13 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:13.828553 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.814151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.14.153.62:45477)
0619 16:49:13.814685 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:13.814690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:13.814763 (+ 73us) server_negotiation.cc:407] Connection header received
0619 16:49:13.814964 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:13.814969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:13.815122 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:13.815237 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:13.816570 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.817351 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.818736 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.819224 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.819322 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:13.823277 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:13.823282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:13.823285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:13.823319 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:13.825884 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:13.826590 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:13.826595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:13.826597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:13.826719 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:13.827102 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:13.827108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:13.827111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:13.827399 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:13.827645 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:13.827953 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 16:49:13.828191 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":288,"threads_started":1}
I20260619 16:49:13.832420 16686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:13.833079 16686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:13.840334 16962 tablet_service.cc:1467] Processing CreateTablet for tablet fc2f6e5240034e66b679191d5770ca11 (DEFAULT_TABLE table=test-table [id=2ea282323de448aea0bb8b9d7228d2d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:13.840334 16823 tablet_service.cc:1467] Processing CreateTablet for tablet fc2f6e5240034e66b679191d5770ca11 (DEFAULT_TABLE table=test-table [id=2ea282323de448aea0bb8b9d7228d2d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:13.840456 17101 tablet_service.cc:1467] Processing CreateTablet for tablet fc2f6e5240034e66b679191d5770ca11 (DEFAULT_TABLE table=test-table [id=2ea282323de448aea0bb8b9d7228d2d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:13.840770 16962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2f6e5240034e66b679191d5770ca11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:13.840775 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2f6e5240034e66b679191d5770ca11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:13.840972 17101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2f6e5240034e66b679191d5770ca11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:13.846423 17281 tablet_bootstrap.cc:492] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: Bootstrap starting.
I20260619 16:49:13.848047 17283 tablet_bootstrap.cc:492] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: Bootstrap starting.
I20260619 16:49:13.848393 17281 tablet_bootstrap.cc:654] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:13.850410 17283 tablet_bootstrap.cc:654] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:13.851809 17282 tablet_bootstrap.cc:492] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Bootstrap starting.
I20260619 16:49:13.854472 17282 tablet_bootstrap.cc:654] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:13.855710 17281 tablet_bootstrap.cc:492] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: No bootstrap required, opened a new log
I20260619 16:49:13.855896 17281 ts_tablet_manager.cc:1397] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 16:49:13.856627 17281 raft_consensus.cc:348] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:13.856841 17281 raft_consensus.cc:374] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:13.857169 17281 raft_consensus.cc:729] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c31cda70ae4130b41a033dcee32f1f, State: Initialized, Role: FOLLOWER
I20260619 16:49:13.857451 17281 consensus_queue.cc:260] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:13.858762 17281 ts_tablet_manager.cc:1428] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:13.859445 17283 tablet_bootstrap.cc:492] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: No bootstrap required, opened a new log
I20260619 16:49:13.859617 17283 ts_tablet_manager.cc:1397] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260619 16:49:13.860255 17282 tablet_bootstrap.cc:492] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: No bootstrap required, opened a new log
I20260619 16:49:13.860430 17282 ts_tablet_manager.cc:1397] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260619 16:49:13.860491 17283 raft_consensus.cc:348] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:13.860745 17283 raft_consensus.cc:374] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:13.860844 17283 raft_consensus.cc:729] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac707b8fc4c44ede8d9d1975fc1e1220, State: Initialized, Role: FOLLOWER
I20260619 16:49:13.861120 17283 consensus_queue.cc:260] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:13.861223 17282 raft_consensus.cc:348] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:13.861452 17282 raft_consensus.cc:374] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:13.861533 17282 raft_consensus.cc:729] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6fbfbc814547f5a8e928ee2ebdbf49, State: Initialized, Role: FOLLOWER
I20260619 16:49:13.861768 17282 consensus_queue.cc:260] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:13.862280 17283 ts_tablet_manager.cc:1428] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 16:49:13.863178 17282 ts_tablet_manager.cc:1428] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260619 16:49:13.964287 17287 raft_consensus.cc:482] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:13.964548 17287 raft_consensus.cc:504] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:13.966030 17287 leader_election.cc:288] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353), ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981)
I20260619 16:49:13.984740 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.966306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:45981 (local address 127.14.153.2:55893)
0619 16:49:13.978830 (+ 12524us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:13.978855 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:13.979046 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:13.979572 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:13.979577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:13.979611 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:13.979949 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:13.979958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.981070 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.981077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:13.982575 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:13.982586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.983873 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.983879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:13.984044 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:13.984067 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:49:13.984190 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:49:13.984331 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12403,"mutex_wait_us":47,"thread_start_us":13019,"threads_started":1}
I20260619 16:49:13.984709 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.966477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:55893 (local address 127.14.153.3:45981)
0619 16:49:13.979288 (+ 12811us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:13.979292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:13.979311 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:49:13.979367 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:13.979371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:13.979424 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:13.979546 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:13.980127 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.980932 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.982735 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.983682 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.983861 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:13.983934 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:13.984215 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:49:13.984342 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12682,"thread_start_us":12510,"threads_started":1}
I20260619 16:49:13.985110 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.966452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:48345 (local address 127.14.153.1:38353)
0619 16:49:13.978594 (+ 12142us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:13.978601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:13.979752 (+ 1151us) server_negotiation.cc:407] Connection header received
0619 16:49:13.979972 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:13.979977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:13.980044 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:13.980153 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:13.980699 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.981457 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.983194 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.984050 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.984273 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:13.984343 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:13.984671 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 16:49:13.984797 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11585,"thread_start_us":11395,"threads_started":1}
I20260619 16:49:13.985140 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.966306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:38353 (local address 127.14.153.2:48345)
0619 16:49:13.979622 (+ 13316us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:13.979638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:13.979823 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:13.980169 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:13.980174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:13.980198 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:13.980520 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:13.980527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.981595 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.981603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:13.982980 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:13.982991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.984274 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.984280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:13.984459 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:13.984501 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:49:13.984607 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:49:13.984738 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13181,"mutex_wait_us":40,"thread_start_us":12266,"threads_started":1}
I20260619 16:49:13.985872 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2f6e5240034e66b679191d5770ca11" candidate_uuid: "a1c31cda70ae4130b41a033dcee32f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" is_pre_election: true
I20260619 16:49:13.986092 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2f6e5240034e66b679191d5770ca11" candidate_uuid: "a1c31cda70ae4130b41a033dcee32f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" is_pre_election: true
I20260619 16:49:13.986200 16847 raft_consensus.cc:2454] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1c31cda70ae4130b41a033dcee32f1f in term 0.
I20260619 16:49:13.986362 17121 raft_consensus.cc:2454] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1c31cda70ae4130b41a033dcee32f1f in term 0.
I20260619 16:49:13.986726 16916 leader_election.cc:302] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6fbfbc814547f5a8e928ee2ebdbf49, a1c31cda70ae4130b41a033dcee32f1f; no voters:
I20260619 16:49:13.987145 17287 raft_consensus.cc:2781] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:13.987262 17287 raft_consensus.cc:482] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:13.987807 17287 raft_consensus.cc:3037] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:13.989880 17287 raft_consensus.cc:504] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:13.990509 17287 leader_election.cc:288] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [CANDIDATE]: Term 1 election: Requested vote from peers 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353), ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981)
I20260619 16:49:13.991011 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2f6e5240034e66b679191d5770ca11" candidate_uuid: "a1c31cda70ae4130b41a033dcee32f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49"
I20260619 16:49:13.991010 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2f6e5240034e66b679191d5770ca11" candidate_uuid: "a1c31cda70ae4130b41a033dcee32f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220"
I20260619 16:49:13.991241 16847 raft_consensus.cc:3037] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:13.991284 17121 raft_consensus.cc:3037] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:13.993208 16847 raft_consensus.cc:2454] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1c31cda70ae4130b41a033dcee32f1f in term 1.
I20260619 16:49:13.993229 17121 raft_consensus.cc:2454] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1c31cda70ae4130b41a033dcee32f1f in term 1.
I20260619 16:49:13.993677 16916 leader_election.cc:302] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5b6fbfbc814547f5a8e928ee2ebdbf49, a1c31cda70ae4130b41a033dcee32f1f, ac707b8fc4c44ede8d9d1975fc1e1220; no voters:
I20260619 16:49:13.993993 17287 raft_consensus.cc:2781] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:13.994426 17287 raft_consensus.cc:686] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 1 LEADER]: Becoming Leader. State: Replica: a1c31cda70ae4130b41a033dcee32f1f, State: Running, Role: LEADER
I20260619 16:49:13.994871 17287 consensus_queue.cc:237] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:13.998227 16686 catalog_manager.cc:5495] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2). New cstate: current_term: 1 leader_uuid: "a1c31cda70ae4130b41a033dcee32f1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:14.020232 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:14.012752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.14.153.2:40119)
0619 16:49:14.013409 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:14.013414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:14.013433 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:49:14.013509 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:14.013512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:14.013581 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:14.013726 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:14.014135 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.015372 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.017269 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.017684 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.017777 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:14.018341 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:14.019166 (+ 825us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:14.019729 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:14.019776 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 16:49:14.019867 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":431,"threads_started":1}
I20260619 16:49:14.035996 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:14.030040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:55355 (local address 127.14.153.3:45981)
0619 16:49:14.030255 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:14.030261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:14.030395 (+ 134us) server_negotiation.cc:407] Connection header received
0619 16:49:14.030569 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:14.030575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:14.030662 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:14.030752 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:14.031678 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.032751 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.034230 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.034850 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.035017 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:14.035455 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:14.035561 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:14.035647 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:14.035754 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:49:14.035810 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:49:14.036214 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:14.029757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:45981 (local address 127.14.153.2:55355)
0619 16:49:14.030311 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:14.030326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:14.030434 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:14.030782 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:14.030787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:14.030805 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:14.031534 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:14.031545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.032910 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.032915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:14.034083 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:14.034094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.035077 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.035086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:14.035240 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:14.035311 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:14.035667 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:14.035676 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:49:14.035740 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:49:14.035836 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":355,"threads_started":1}
I20260619 16:49:14.043917 16847 raft_consensus.cc:1261] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Refusing update from remote peer a1c31cda70ae4130b41a033dcee32f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:14.043960 17121 raft_consensus.cc:1261] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Refusing update from remote peer a1c31cda70ae4130b41a033dcee32f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:14.044560 17293 consensus_queue.cc:1035] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:14.044921 17287 consensus_queue.cc:1035] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:14.049468 17254 mvcc.cc:205] Tried to move back new op lower bound from 7298612240558297088 to 7298612240367296512. Current Snapshot: MvccSnapshot[applied={T|T < 7298612240558297088}]
I20260619 16:49:14.077222 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:14.068186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:44425 (local address 127.14.153.2:40119)
0619 16:49:14.068464 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:14.068470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:14.071664 (+ 3194us) server_negotiation.cc:407] Connection header received
0619 16:49:14.071831 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:14.071837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:14.071918 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:14.072015 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:14.072463 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.073564 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.075279 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.075948 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.076072 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:14.076510 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:14.076647 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:14.076738 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:14.076862 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 16:49:14.076936 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260619 16:49:14.077801 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:14.068002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:40119 (local address 127.14.153.3:44425)
0619 16:49:14.071554 (+ 3552us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:14.071571 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:14.071703 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:14.072035 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:14.072038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:14.072047 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:14.072323 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:14.072331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.073707 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.073710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:14.075118 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:14.075133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.076159 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.076164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:14.076301 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:14.076340 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:14.076745 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:14.076753 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:49:14.076824 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:49:14.076923 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3415,"thread_start_us":3240,"threads_started":1}
I20260619 16:49:14.109243 16686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:14.109496 16686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:14.112782 16686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401: Sending DeleteTablet for 3 replicas of tablet fc2f6e5240034e66b679191d5770ca11
I20260619 16:49:14.113613 16823 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2f6e5240034e66b679191d5770ca11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:14 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:34858
I20260619 16:49:14.113692 17101 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2f6e5240034e66b679191d5770ca11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:14 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260619 16:49:14.113698 16962 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2f6e5240034e66b679191d5770ca11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:14 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:51480
I20260619 16:49:14.113977 17270 tablet_replica.cc:330] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: stopping tablet replica
I20260619 16:49:14.113978 17272 tablet_replica.cc:330] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: stopping tablet replica
I20260619 16:49:14.114192 17270 raft_consensus.cc:2229] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:14.114213 17272 raft_consensus.cc:2229] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:14.114384 17270 raft_consensus.cc:2258] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:14.114691 17271 tablet_replica.cc:330] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: stopping tablet replica
I20260619 16:49:14.114709 17272 raft_consensus.cc:2258] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:14.114924 17271 raft_consensus.cc:2229] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:14.115159 17271 raft_consensus.cc:2258] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:14.115952 17272 ts_tablet_manager.cc:1905] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:14.116225 17270 ts_tablet_manager.cc:1905] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:14.116273 17271 ts_tablet_manager.cc:1905] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:14.123529 17272 ts_tablet_manager.cc:1918] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:14.123529 17271 ts_tablet_manager.cc:1918] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:14.123694 17271 log.cc:1196] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/fc2f6e5240034e66b679191d5770ca11
I20260619 16:49:14.123986 17272 log.cc:1196] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/fc2f6e5240034e66b679191d5770ca11
I20260619 16:49:14.124142 17271 ts_tablet_manager.cc:1939] T fc2f6e5240034e66b679191d5770ca11 P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting consensus metadata
I20260619 16:49:14.124505 17272 ts_tablet_manager.cc:1939] T fc2f6e5240034e66b679191d5770ca11 P a1c31cda70ae4130b41a033dcee32f1f: Deleting consensus metadata
I20260619 16:49:14.125298 16671 catalog_manager.cc:4841] TS ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981): tablet fc2f6e5240034e66b679191d5770ca11 (table test-table [id=2ea282323de448aea0bb8b9d7228d2d4]) successfully deleted
I20260619 16:49:14.125277 16686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:49:14.125564 16673 catalog_manager.cc:4841] TS a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119): tablet fc2f6e5240034e66b679191d5770ca11 (table test-table [id=2ea282323de448aea0bb8b9d7228d2d4]) successfully deleted
W20260619 16:49:14.126108 16686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:14.127529 17270 ts_tablet_manager.cc:1918] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:14.127696 17270 log.cc:1196] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/fc2f6e5240034e66b679191d5770ca11
I20260619 16:49:14.128135 17270 ts_tablet_manager.cc:1939] T fc2f6e5240034e66b679191d5770ca11 P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting consensus metadata
I20260619 16:49:14.129180 16673 catalog_manager.cc:4841] TS 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353): tablet fc2f6e5240034e66b679191d5770ca11 (table test-table [id=2ea282323de448aea0bb8b9d7228d2d4]) successfully deleted
I20260619 16:49:14.134264 16823 tablet_service.cc:1467] Processing CreateTablet for tablet 34084cd2277341bc860a7a521c1236af (DEFAULT_TABLE table=test-table [id=6f5b5afbc33849f982e5475b0a39f730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:14.134268 16962 tablet_service.cc:1467] Processing CreateTablet for tablet 34084cd2277341bc860a7a521c1236af (DEFAULT_TABLE table=test-table [id=6f5b5afbc33849f982e5475b0a39f730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:14.134263 17101 tablet_service.cc:1467] Processing CreateTablet for tablet 34084cd2277341bc860a7a521c1236af (DEFAULT_TABLE table=test-table [id=6f5b5afbc33849f982e5475b0a39f730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:14.134716 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34084cd2277341bc860a7a521c1236af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:14.134721 17101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34084cd2277341bc860a7a521c1236af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:14.134727 16962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34084cd2277341bc860a7a521c1236af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:14.141204 17283 tablet_bootstrap.cc:492] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: Bootstrap starting.
I20260619 16:49:14.141232 17282 tablet_bootstrap.cc:492] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: Bootstrap starting.
I20260619 16:49:14.143185 17283 tablet_bootstrap.cc:654] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:14.143327 17282 tablet_bootstrap.cc:654] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:14.144851 17283 tablet_bootstrap.cc:492] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: No bootstrap required, opened a new log
I20260619 16:49:14.145047 17283 ts_tablet_manager.cc:1397] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 16:49:14.145797 17283 raft_consensus.cc:348] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:14.146060 17283 raft_consensus.cc:374] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:14.146139 17283 raft_consensus.cc:729] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac707b8fc4c44ede8d9d1975fc1e1220, State: Initialized, Role: FOLLOWER
I20260619 16:49:14.146384 17283 consensus_queue.cc:260] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:14.146903 17283 ts_tablet_manager.cc:1428] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 16:49:14.148028 17281 tablet_bootstrap.cc:492] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: Bootstrap starting.
I20260619 16:49:14.149823 17281 tablet_bootstrap.cc:654] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:14.153338 17282 tablet_bootstrap.cc:492] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: No bootstrap required, opened a new log
I20260619 16:49:14.153543 17282 ts_tablet_manager.cc:1397] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260619 16:49:14.154255 17281 tablet_bootstrap.cc:492] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: No bootstrap required, opened a new log
I20260619 16:49:14.154497 17281 ts_tablet_manager.cc:1397] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260619 16:49:14.154377 17282 raft_consensus.cc:348] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:14.154649 17282 raft_consensus.cc:374] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:14.154731 17282 raft_consensus.cc:729] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6fbfbc814547f5a8e928ee2ebdbf49, State: Initialized, Role: FOLLOWER
I20260619 16:49:14.154986 17282 consensus_queue.cc:260] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:14.155552 17282 ts_tablet_manager.cc:1428] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:14.155746 17281 raft_consensus.cc:348] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:14.156107 17281 raft_consensus.cc:374] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:14.156244 17281 raft_consensus.cc:729] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c31cda70ae4130b41a033dcee32f1f, State: Initialized, Role: FOLLOWER
I20260619 16:49:14.156498 17281 consensus_queue.cc:260] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:14.157123 17281 ts_tablet_manager.cc:1428] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:14.429913 17288 raft_consensus.cc:482] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:14.430200 17288 raft_consensus.cc:504] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:14.431177 17288 leader_election.cc:288] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119), ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981)
I20260619 16:49:14.431784 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34084cd2277341bc860a7a521c1236af" candidate_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" is_pre_election: true
I20260619 16:49:14.431818 16982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34084cd2277341bc860a7a521c1236af" candidate_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c31cda70ae4130b41a033dcee32f1f" is_pre_election: true
I20260619 16:49:14.432041 17121 raft_consensus.cc:2454] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b6fbfbc814547f5a8e928ee2ebdbf49 in term 0.
I20260619 16:49:14.432068 16982 raft_consensus.cc:2454] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b6fbfbc814547f5a8e928ee2ebdbf49 in term 0.
I20260619 16:49:14.432508 16774 leader_election.cc:302] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6fbfbc814547f5a8e928ee2ebdbf49, ac707b8fc4c44ede8d9d1975fc1e1220; no voters:
I20260619 16:49:14.432860 17288 raft_consensus.cc:2781] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:14.432978 17288 raft_consensus.cc:482] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:14.433075 17288 raft_consensus.cc:3037] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:14.435376 17288 raft_consensus.cc:504] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:14.436129 17288 leader_election.cc:288] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [CANDIDATE]: Term 1 election: Requested vote from peers a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119), ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981)
I20260619 16:49:14.436510 16982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34084cd2277341bc860a7a521c1236af" candidate_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c31cda70ae4130b41a033dcee32f1f"
I20260619 16:49:14.436514 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34084cd2277341bc860a7a521c1236af" candidate_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220"
I20260619 16:49:14.436724 16982 raft_consensus.cc:3037] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:14.436766 17121 raft_consensus.cc:3037] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:14.439093 16982 raft_consensus.cc:2454] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b6fbfbc814547f5a8e928ee2ebdbf49 in term 1.
I20260619 16:49:14.439586 16776 leader_election.cc:302] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6fbfbc814547f5a8e928ee2ebdbf49, a1c31cda70ae4130b41a033dcee32f1f; no voters:
I20260619 16:49:14.439764 17121 raft_consensus.cc:2454] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b6fbfbc814547f5a8e928ee2ebdbf49 in term 1.
I20260619 16:49:14.439970 17288 raft_consensus.cc:2781] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:14.440658 17288 raft_consensus.cc:686] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 LEADER]: Becoming Leader. State: Replica: 5b6fbfbc814547f5a8e928ee2ebdbf49, State: Running, Role: LEADER
I20260619 16:49:14.440948 17288 consensus_queue.cc:237] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } }
I20260619 16:49:14.443805 16686 catalog_manager.cc:5495] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1). New cstate: current_term: 1 leader_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b6fbfbc814547f5a8e928ee2ebdbf49" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 38353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:14.533751 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:14.524076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.14.153.1:38353)
0619 16:49:14.524639 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:14.524645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:14.528431 (+ 3786us) server_negotiation.cc:407] Connection header received
0619 16:49:14.528642 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:14.528647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:14.528726 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:14.528835 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:14.529287 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.530210 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.531721 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:14.532112 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:14.532257 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:14.532640 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:14.532737 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:14.532838 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:14.532972 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 16:49:14.533043 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":297,"threads_started":1}
I20260619 16:49:14.542829 16982 raft_consensus.cc:1261] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Refusing update from remote peer 5b6fbfbc814547f5a8e928ee2ebdbf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:14.542954 17121 raft_consensus.cc:1261] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Refusing update from remote peer 5b6fbfbc814547f5a8e928ee2ebdbf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:14.543457 17288 consensus_queue.cc:1035] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1c31cda70ae4130b41a033dcee32f1f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:14.543743 17309 consensus_queue.cc:1035] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac707b8fc4c44ede8d9d1975fc1e1220" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:49:14.567183 17255 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:49:16.175612 16686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:16.176219 16686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:16.180330 16686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e440e40df306459db37713d227ef7401: Sending DeleteTablet for 3 replicas of tablet 34084cd2277341bc860a7a521c1236af
I20260619 16:49:16.181237 16823 tablet_service.cc:1514] Processing DeleteTablet for tablet 34084cd2277341bc860a7a521c1236af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:16 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:34858
I20260619 16:49:16.181274 16962 tablet_service.cc:1514] Processing DeleteTablet for tablet 34084cd2277341bc860a7a521c1236af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:16 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:51480
I20260619 16:49:16.181366 17101 tablet_service.cc:1514] Processing DeleteTablet for tablet 34084cd2277341bc860a7a521c1236af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:16 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260619 16:49:16.182017 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16757
I20260619 16:49:16.193797 17343 tablet_replica.cc:330] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: stopping tablet replica
I20260619 16:49:16.194278 17344 tablet_replica.cc:330] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: stopping tablet replica
I20260619 16:49:16.194316 17343 raft_consensus.cc:2229] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:16.194689 17344 raft_consensus.cc:2229] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:16.194861 17343 raft_consensus.cc:2258] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:16.195107 17344 raft_consensus.cc:2258] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:16.196393 17343 ts_tablet_manager.cc:1905] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:16.196856 17344 ts_tablet_manager.cc:1905] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:16.204738 17344 ts_tablet_manager.cc:1918] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:16.204973 17344 log.cc:1196] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/34084cd2277341bc860a7a521c1236af
I20260619 16:49:16.205530 17344 ts_tablet_manager.cc:1939] T 34084cd2277341bc860a7a521c1236af P ac707b8fc4c44ede8d9d1975fc1e1220: Deleting consensus metadata
I20260619 16:49:16.205966 17343 ts_tablet_manager.cc:1918] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:16.206566 17342 tablet_replica.cc:330] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: stopping tablet replica
I20260619 16:49:16.206745 17343 log.cc:1196] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/34084cd2277341bc860a7a521c1236af
I20260619 16:49:16.206730 16671 catalog_manager.cc:4841] TS ac707b8fc4c44ede8d9d1975fc1e1220 (127.14.153.3:45981): tablet 34084cd2277341bc860a7a521c1236af (table test-table [id=6f5b5afbc33849f982e5475b0a39f730]) successfully deleted
I20260619 16:49:16.206815 17342 raft_consensus.cc:2229] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:16.207562 17343 ts_tablet_manager.cc:1939] T 34084cd2277341bc860a7a521c1236af P 5b6fbfbc814547f5a8e928ee2ebdbf49: Deleting consensus metadata
I20260619 16:49:16.207554 17342 raft_consensus.cc:2258] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:16.208639 17342 ts_tablet_manager.cc:1905] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:16.208771 16673 catalog_manager.cc:4841] TS 5b6fbfbc814547f5a8e928ee2ebdbf49 (127.14.153.1:38353): tablet 34084cd2277341bc860a7a521c1236af (table test-table [id=6f5b5afbc33849f982e5475b0a39f730]) successfully deleted
I20260619 16:49:16.213426 17342 ts_tablet_manager.cc:1918] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:16.213593 17342 log.cc:1196] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/34084cd2277341bc860a7a521c1236af
I20260619 16:49:16.213976 17342 ts_tablet_manager.cc:1939] T 34084cd2277341bc860a7a521c1236af P a1c31cda70ae4130b41a033dcee32f1f: Deleting consensus metadata
I20260619 16:49:16.215004 16673 catalog_manager.cc:4841] TS a1c31cda70ae4130b41a033dcee32f1f (127.14.153.2:40119): tablet 34084cd2277341bc860a7a521c1236af (table test-table [id=6f5b5afbc33849f982e5475b0a39f730]) successfully deleted
Jun 19 16:49:16 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:16.222522 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.183668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.14.153.1:38353)
0619 16:49:16.201304 (+ 17636us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.201310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.201338 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:49:16.201402 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.201407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.201507 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.203467 (+ 1960us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.203681 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.204769 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.206586 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.207731 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.207869 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.216413 (+ 8544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:16.216429 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:16.216445 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:16.216484 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:16.219572 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.220346 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.220354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.220356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.220492 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.220910 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.220920 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.220922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.221273 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:16.221580 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.221909 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.222150 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17480,"thread_start_us":18006,"threads_started":1}
W20260619 16:49:16.223253 16888 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35634
W20260619 16:49:16.223945 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16757: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:49:16.224189 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 16757
I20260619 16:49:16.246776 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16897
Jun 19 16:49:16 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:49:16.262756 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.247668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.14.153.2:40119)
0619 16:49:16.248303 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.248307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.248324 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:49:16.248376 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.248379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.248435 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.248528 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.249852 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.250590 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.252106 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.252494 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.252608 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.256742 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:16.256758 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:16.256775 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:16.256816 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:16.259896 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.260649 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.260656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.260659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.260855 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.261252 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.261257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.261267 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.261618 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:16.261844 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.262181 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.262389 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":313,"threads_started":1}
W20260619 16:49:16.263561 17027 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44794
W20260619 16:49:16.264195 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16897: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:49:16.264385 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 16897
I20260619 16:49:16.286684 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17036
Jun 19 16:49:16 dist-test-slave-qb36 krb5kdc[16641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:49:16.302171 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.287474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.14.153.3:45981)
0619 16:49:16.287970 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.287975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.287990 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:49:16.288043 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.288047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.288108 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.288204 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.289575 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.290311 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.291697 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.292084 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.292179 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.296669 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:16.296684 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:16.296701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:16.296734 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:16.299487 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.300160 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.300168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.300171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.300297 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.300663 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.300670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.300673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.301004 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:16.301304 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.301606 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.301834 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":268,"threads_started":1}
W20260619 16:49:16.302795 17166 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47420
W20260619 16:49:16.303472 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17036: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:49:16.303669 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 17036
I20260619 16:49:16.328438 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 16654
I20260619 16:49:16.329711 16726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:16.443338 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 16654
2026-06-19T16:49:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8744 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:16 dist-test-slave-qb36 krb5kdc[17352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:16 dist-test-slave-qb36 krb5kdc[17352](info): set up 2 sockets
Jun 19 16:49:16 dist-test-slave-qb36 krb5kdc[17352](info): commencing operation
krb5kdc: starting...
W20260619 16:49:17.532181 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:17 dist-test-slave-qb36 krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:49:17.722983 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42083
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:18.049135 17365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:18.049566 17365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:18.049729 17365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:18.057184 17365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:18.057350 17365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:18.057469 17365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:18.057565 17365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:18.057646 17365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:18.067823 17365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:18.069223 17365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:18.070716 17365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:18.083725 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:18.083648 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:18.083902 17365 server_base.cc:1034] running on GCE node
W20260619 16:49:18.083621 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:18.085022 17365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:18.086501 17365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:18.087743 17365 hybrid_clock.cc:648] HybridClock initialized: now 1781887758087688 us; error 39 us; skew 500 ppm
I20260619 16:49:18.088212 17365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:18 dist-test-slave-qb36 krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887758, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:18.091391 17365 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:49:18.092973 17365 webserver.cc:460] Webserver started at http://127.14.153.62:41489/ using document root <none> and password file <none>
I20260619 16:49:18.093710 17365 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:18.093858 17365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:18.094239 17365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:18.097327 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "1b659c185e82401e92b3162ad7fa680f"
format_stamp: "Formatted at 2026-06-19 16:49:18 on dist-test-slave-qb36"
server_key: "d9880617f60b5f0e0927c9196418edd4"
server_key_iv: "ddab5881fb73b2814ff555a93e09bd49"
server_key_version: "encryptionkey@0"
I20260619 16:49:18.098124 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "1b659c185e82401e92b3162ad7fa680f"
format_stamp: "Formatted at 2026-06-19 16:49:18 on dist-test-slave-qb36"
server_key: "d9880617f60b5f0e0927c9196418edd4"
server_key_iv: "ddab5881fb73b2814ff555a93e09bd49"
server_key_version: "encryptionkey@0"
I20260619 16:49:18.103647 17365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:49:18.107314 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:18.108758 17365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:18.108942 17365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "1b659c185e82401e92b3162ad7fa680f"
format_stamp: "Formatted at 2026-06-19 16:49:18 on dist-test-slave-qb36"
server_key: "d9880617f60b5f0e0927c9196418edd4"
server_key_iv: "ddab5881fb73b2814ff555a93e09bd49"
server_key_version: "encryptionkey@0"
I20260619 16:49:18.109119 17365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:18.129060 17365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:18.133589 17365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:18.133836 17365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:18.153311 17365 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:42083
I20260619 16:49:18.153342 17431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:42083 every 8 connection(s)
I20260619 16:49:18.154903 17365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:18.158711 17432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:18.161945 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 17365
I20260619 16:49:18.162186 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:18.162941 14948 external_mini_cluster.cc:1428] Setting key f3a22c3ddc217524230de3334e32c7fe
I20260619 16:49:18.171324 17432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f: Bootstrap starting.
I20260619 16:49:18.174589 17432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:18.175705 17432 log.cc:824] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:49:18 dist-test-slave-qb36 krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:18.178841 17432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f: No bootstrap required, opened a new log
I20260619 16:49:18.185354 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:18.165146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.14.153.62:42083)
0619 16:49:18.165908 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:18.165920 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:18.165963 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:49:18.167716 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:18.167741 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:18.168166 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:18.168595 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:18.170052 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.171481 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.172764 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.173442 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.173647 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:18.177800 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:18.177808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:18.177824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:18.177862 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:18.181063 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:18.181802 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:18.181818 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:18.181824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:18.181984 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:18.182395 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:18.182399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:18.182401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:18.183021 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:18.183393 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:18.183844 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 16:49:18.184285 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":361,"threads_started":1}
I20260619 16:49:18.186952 17432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b659c185e82401e92b3162ad7fa680f" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42083 } }
I20260619 16:49:18.187384 17432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:18.187494 17432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b659c185e82401e92b3162ad7fa680f, State: Initialized, Role: FOLLOWER
I20260619 16:49:18.188098 17432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b659c185e82401e92b3162ad7fa680f" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42083 } }
I20260619 16:49:18.188352 17432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:18.188535 17432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:18.188704 17432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:18.190943 17432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b659c185e82401e92b3162ad7fa680f" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42083 } }
I20260619 16:49:18.191498 17432 leader_election.cc:302] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b659c185e82401e92b3162ad7fa680f; no voters:
I20260619 16:49:18.192071 17432 leader_election.cc:288] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:18.192225 17437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:18.192867 17437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [term 1 LEADER]: Becoming Leader. State: Replica: 1b659c185e82401e92b3162ad7fa680f, State: Running, Role: LEADER
I20260619 16:49:18.193396 17437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b659c185e82401e92b3162ad7fa680f" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42083 } }
I20260619 16:49:18.194165 17432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:18.196151 17439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b659c185e82401e92b3162ad7fa680f. Latest consensus state: current_term: 1 leader_uuid: "1b659c185e82401e92b3162ad7fa680f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b659c185e82401e92b3162ad7fa680f" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42083 } } }
I20260619 16:49:18.196585 17439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:18.196514 17438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b659c185e82401e92b3162ad7fa680f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b659c185e82401e92b3162ad7fa680f" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42083 } } }
I20260619 16:49:18.196861 17438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:18.198645 17442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:18.204031 17442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:18.213166 17442 catalog_manager.cc:1347] Generated new cluster ID: ab32efd127754558ac7128bd4fea9526
I20260619 16:49:18.213326 17442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:18.238269 17442 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:18.239816 17442 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:18.253938 17442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b659c185e82401e92b3162ad7fa680f: Generated new TSK 0
I20260619 16:49:18.254850 17442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:49:18.298796 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42083
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:18.630934 17458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:18.631309 17458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:18.631403 17458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:18.638743 17458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:18.638859 17458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:18.638983 17458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:18.649106 17458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:18.650316 17458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:18.651684 17458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:18.664950 17463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:18.665032 17464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:18.665011 17466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:18.665774 17458 server_base.cc:1034] running on GCE node
I20260619 16:49:18.666455 17458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:18.667572 17458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:18.668779 17458 hybrid_clock.cc:648] HybridClock initialized: now 1781887758668731 us; error 43 us; skew 500 ppm
I20260619 16:49:18.669131 17458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:18 dist-test-slave-qb36 krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887758, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:18.672501 17458 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:49:18.673995 17458 webserver.cc:460] Webserver started at http://127.14.153.1:46125/ using document root <none> and password file <none>
I20260619 16:49:18.674702 17458 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:18.674844 17458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:18.675227 17458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:18.678148 17458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "08f80cd60e6c4244aad4162abd58ee7c"
format_stamp: "Formatted at 2026-06-19 16:49:18 on dist-test-slave-qb36"
server_key: "47dac6272cc6fba69ea55fff7bc2ff0c"
server_key_iv: "e7bd0fd258d0f6af30de916dc5ea8a49"
server_key_version: "encryptionkey@0"
I20260619 16:49:18.678922 17458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "08f80cd60e6c4244aad4162abd58ee7c"
format_stamp: "Formatted at 2026-06-19 16:49:18 on dist-test-slave-qb36"
server_key: "47dac6272cc6fba69ea55fff7bc2ff0c"
server_key_iv: "e7bd0fd258d0f6af30de916dc5ea8a49"
server_key_version: "encryptionkey@0"
I20260619 16:49:18.684553 17458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:49:18.688196 17473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:18.689596 17458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:49:18.689775 17458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "08f80cd60e6c4244aad4162abd58ee7c"
format_stamp: "Formatted at 2026-06-19 16:49:18 on dist-test-slave-qb36"
server_key: "47dac6272cc6fba69ea55fff7bc2ff0c"
server_key_iv: "e7bd0fd258d0f6af30de916dc5ea8a49"
server_key_version: "encryptionkey@0"
I20260619 16:49:18.689980 17458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:18.707996 17458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:18.712009 17458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:18.712298 17458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:18.713322 17458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:18.714740 17458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:18.714836 17458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:18.714988 17458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:18.715083 17458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:18.747967 17458 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:36233
I20260619 16:49:18.748061 17585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:36233 every 8 connection(s)
I20260619 16:49:18.749301 17458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:18.757689 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 17458
I20260619 16:49:18.757932 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:18.758659 14948 external_mini_cluster.cc:1428] Setting key 6df0ec0d06ecd18cb48f75d551e8d526
Jun 19 16:49:18 dist-test-slave-qb36 krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887758, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:18.771374 17588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:18.751920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:55753 (local address 127.14.153.62:42083)
0619 16:49:18.752371 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:18.752375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:18.753286 (+ 911us) server_negotiation.cc:407] Connection header received
0619 16:49:18.755235 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:18.755240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:18.755336 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:18.755456 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:18.757091 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.758258 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.760081 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.760697 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.760817 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:18.765161 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:18.765167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:18.765170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:18.765202 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:18.767845 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:18.768863 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:18.768868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:18.768870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:18.768989 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:18.769493 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:18.769496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:18.769498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:18.769753 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:18.769859 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:18.770785 (+ 926us) server_negotiation.cc:299] Negotiation successful
0619 16:49:18.770953 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":244,"threads_started":1}
I20260619 16:49:18.773053 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:18.752372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42083 (local address 127.14.153.1:55753)
0619 16:49:18.753118 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:18.753163 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:18.754881 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:18.755717 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:18.755726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:18.756267 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:18.756908 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:18.756928 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.758420 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.758424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:18.759932 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:18.759944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.760941 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.760950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:18.761248 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:18.762044 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:18.762086 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:18.764886 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:18.768001 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:18.768015 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:18.768037 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:18.768711 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:18.769105 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:18.769113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:18.769120 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:18.769379 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:18.769874 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:18.769885 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:18.770377 (+ 492us) client_negotiation.cc:769] Sending connection context
0619 16:49:18.770708 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 16:49:18.771376 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":265,"threads_started":1}
I20260619 16:49:18.774766 17586 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42083
I20260619 16:49:18.775143 17586 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:18.775864 17586 heartbeater.cc:511] Master 127.14.153.62:42083 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:18.778007 17397 ts_manager.cc:194] Registered new tserver with Master: 08f80cd60e6c4244aad4162abd58ee7c (127.14.153.1:36233)
I20260619 16:49:18.779816 17397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:55753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:49:18.806017 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42083
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:19.134806 17592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:19.135248 17592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:19.135429 17592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:19.143478 17592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:19.143653 17592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:19.143849 17592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:19.154353 17592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:19.155702 17592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:19.157070 17592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:19.171198 17600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.171168 17597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.171244 17598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:19.171927 17592 server_base.cc:1034] running on GCE node
I20260619 16:49:19.172678 17592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:19.173652 17592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:19.174885 17592 hybrid_clock.cc:648] HybridClock initialized: now 1781887759174822 us; error 58 us; skew 500 ppm
I20260619 16:49:19.175369 17592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:19 dist-test-slave-qb36 krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:19.178526 17592 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:49:19.179997 17592 webserver.cc:460] Webserver started at http://127.14.153.2:39443/ using document root <none> and password file <none>
I20260619 16:49:19.180693 17592 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:19.180790 17592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:19.181074 17592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:19.184103 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "e91b61de3d1e456ab069b5a6c1db491a"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-qb36"
server_key: "b15d08250872f58bc4add9832e85ee7b"
server_key_iv: "a197e7c1f7ce66db05e2773a3a295d4d"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.184862 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "e91b61de3d1e456ab069b5a6c1db491a"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-qb36"
server_key: "b15d08250872f58bc4add9832e85ee7b"
server_key_iv: "a197e7c1f7ce66db05e2773a3a295d4d"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.190213 17592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:49:19.193909 17607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.195524 17592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:19.195698 17592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "e91b61de3d1e456ab069b5a6c1db491a"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-qb36"
server_key: "b15d08250872f58bc4add9832e85ee7b"
server_key_iv: "a197e7c1f7ce66db05e2773a3a295d4d"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.195905 17592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:19.225795 17592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:19.230147 17592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:19.230451 17592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:19.231438 17592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:19.232780 17592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:19.232853 17592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.233001 17592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:19.233057 17592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.265815 17592 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:37157
I20260619 16:49:19.265862 17719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:37157 every 8 connection(s)
I20260619 16:49:19.267158 17592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:19.274714 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 17592
I20260619 16:49:19.274919 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:19.275458 14948 external_mini_cluster.cc:1428] Setting key 9b77220f2258dfa1ee87f3a904afc451
Jun 19 16:49:19 dist-test-slave-qb36 krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:19.287976 17588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.269590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:60713 (local address 127.14.153.62:42083)
0619 16:49:19.269770 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:19.269775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:19.270900 (+ 1125us) server_negotiation.cc:407] Connection header received
0619 16:49:19.272756 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:19.272760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:19.272825 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:19.272903 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:19.274475 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.275609 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.277261 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.277691 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.277822 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.281567 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:19.281573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:19.281576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:19.281610 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:19.284761 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.285564 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.285569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.285577 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.285709 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.286144 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.286151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.286154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.286436 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:19.286582 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:19.287382 (+ 800us) server_negotiation.cc:299] Negotiation successful
0619 16:49:19.287614 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:49:19.289034 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.270069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42083 (local address 127.14.153.2:60713)
0619 16:49:19.270742 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:19.270781 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:19.272486 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:19.273163 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:19.273171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:19.273674 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:19.274303 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.274321 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.275790 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.275794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.277095 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.277107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.277981 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.277994 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.278216 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.278939 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:19.278994 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:19.281318 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:19.284932 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.284942 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.284962 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.285416 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.285821 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.285826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.285830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.286032 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.286572 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:19.286580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:19.286950 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 16:49:19.287281 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 16:49:19.287862 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":272,"threads_started":1}
I20260619 16:49:19.290735 17720 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42083
I20260619 16:49:19.291163 17720 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:19.291932 17720 heartbeater.cc:511] Master 127.14.153.62:42083 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:19.293876 17397 ts_manager.cc:194] Registered new tserver with Master: e91b61de3d1e456ab069b5a6c1db491a (127.14.153.2:37157)
I20260619 16:49:19.294842 17397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:60713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:49:19.320099 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42083
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:19.641674 17725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:19.641966 17725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:19.642129 17725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:19.649359 17725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:19.649471 17725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:19.649654 17725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:19.659564 17725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:19.660892 17725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:19.662186 17725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:19.675563 17733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.675631 17730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.675645 17731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:19.676908 17725 server_base.cc:1034] running on GCE node
I20260619 16:49:19.677548 17725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:19.678548 17725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:19.679778 17725 hybrid_clock.cc:648] HybridClock initialized: now 1781887759679734 us; error 41 us; skew 500 ppm
I20260619 16:49:19.680136 17725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:19 dist-test-slave-qb36 krb5kdc[17352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:19.686324 17725 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:49:19.687997 17725 webserver.cc:460] Webserver started at http://127.14.153.3:36047/ using document root <none> and password file <none>
I20260619 16:49:19.688743 17725 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:19.688885 17725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:19.689205 17725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:19.692272 17725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "2e01895ec03946c893212a1e27644eb9"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-qb36"
server_key: "1d23eee56edde138dc765082ea072cde"
server_key_iv: "388bbc72d4a0a26872b195a0b78e274e"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.693120 17725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "2e01895ec03946c893212a1e27644eb9"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-qb36"
server_key: "1d23eee56edde138dc765082ea072cde"
server_key_iv: "388bbc72d4a0a26872b195a0b78e274e"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.698659 17725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 16:49:19.702438 17740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.703812 17725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:49:19.703979 17725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "2e01895ec03946c893212a1e27644eb9"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-qb36"
server_key: "1d23eee56edde138dc765082ea072cde"
server_key_iv: "388bbc72d4a0a26872b195a0b78e274e"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.704178 17725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:19.717816 17725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:19.721824 17725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:19.722119 17725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:19.723112 17725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:19.724416 17725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:19.724540 17725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.724687 17725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:19.724761 17725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.760565 17725 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:42829
I20260619 16:49:19.760629 17852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:42829 every 8 connection(s)
I20260619 16:49:19.761843 17725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:19.768566 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 17725
I20260619 16:49:19.768779 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:19.769228 14948 external_mini_cluster.cc:1428] Setting key 3709c4cf44f7cb12f65c7aa8c02d06f4
Jun 19 16:49:19 dist-test-slave-qb36 krb5kdc[17352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:19.781899 17588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.764462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:36255 (local address 127.14.153.62:42083)
0619 16:49:19.764664 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:19.764670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:19.765949 (+ 1279us) server_negotiation.cc:407] Connection header received
0619 16:49:19.767805 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:19.767809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:19.767878 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:19.767969 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:19.769609 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.770647 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.772543 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.772932 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.773030 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.776283 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:19.776287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:19.776290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:19.776322 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:19.778828 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.779591 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.779597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.779608 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.779738 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.780188 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.780193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.780197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.780456 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:19.780612 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:19.781326 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 16:49:19.781539 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:49:19.783335 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.764996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42083 (local address 127.14.153.3:36255)
0619 16:49:19.765783 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:19.765824 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:19.767533 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:19.768192 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:19.768200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:19.768749 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:19.769426 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.769445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.770785 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.770788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.772421 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.772433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.773124 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.773129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.773319 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.773927 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:19.773949 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:19.776075 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:19.778945 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.778952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.778965 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.779420 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.779846 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.779854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.779858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.780037 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.780575 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:19.780584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:19.780956 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 16:49:19.781228 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 16:49:19.781758 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":282,"threads_started":1}
I20260619 16:49:19.783855 17586 heartbeater.cc:503] Master 127.14.153.62:42083 was elected leader, sending a full tablet report...
I20260619 16:49:19.785966 17853 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42083
I20260619 16:49:19.786360 17853 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:19.787212 17853 heartbeater.cc:511] Master 127.14.153.62:42083 requested a full tablet report, sending...
I20260619 16:49:19.788926 17397 ts_manager.cc:194] Registered new tserver with Master: 2e01895ec03946c893212a1e27644eb9 (127.14.153.3:42829)
I20260619 16:49:19.790129 17397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:36255
I20260619 16:49:19.797931 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:49:19.840636 17588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:19.837558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.14.153.62:42083)
0619 16:49:19.837791 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:19.837797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:19.838304 (+ 507us) server_negotiation.cc:407] Connection header received
0619 16:49:19.838469 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:19.838475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:19.838570 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:19.838722 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:19.840350 (+ 1628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40618: BlockingRecv error: recv got EOF from 127.0.0.1:40618 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:49:19.844868 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17458
W20260619 16:49:19.847993 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17458: Not authorized: Client connection negotiation failed: client connection to 127.14.153.1:36233: server requires authentication, but client does not have Kerberos credentials available
I20260619 16:49:19.848191 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 17458
I20260619 16:49:19.865525 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17592
W20260619 16:49:19.868302 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17592: Not authorized: Client connection negotiation failed: client connection to 127.14.153.2:37157: server requires authentication, but client does not have Kerberos credentials available
I20260619 16:49:19.868512 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 17592
I20260619 16:49:19.884080 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17725
W20260619 16:49:19.887305 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17725: Not authorized: Client connection negotiation failed: client connection to 127.14.153.3:42829: server requires authentication, but client does not have Kerberos credentials available
I20260619 16:49:19.887439 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 17725
I20260619 16:49:19.903012 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17365
I20260619 16:49:19.904294 17427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:20.006068 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 17365
2026-06-19T16:49:20Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3563 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:20 dist-test-slave-qb36 krb5kdc[17870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:20 dist-test-slave-qb36 krb5kdc[17870](info): set up 2 sockets
Jun 19 16:49:20 dist-test-slave-qb36 krb5kdc[17870](info): commencing operation
krb5kdc: starting...
W20260619 16:49:21.096668 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:21 dist-test-slave-qb36 krb5kdc[17870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:49:21.295194 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34135
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:21.612856 17883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:21.613162 17883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:21.613251 17883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:21.620462 17883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:21.620586 17883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:21.620635 17883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:21.620683 17883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:21.620733 17883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:21.630640 17883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:21.631985 17883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:21.633275 17883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:21.644842 17888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:21.644889 17889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:21.645037 17891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:21.645927 17883 server_base.cc:1034] running on GCE node
I20260619 16:49:21.646668 17883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:21.648165 17883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:21.649375 17883 hybrid_clock.cc:648] HybridClock initialized: now 1781887761649329 us; error 36 us; skew 500 ppm
I20260619 16:49:21.649688 17883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:21 dist-test-slave-qb36 krb5kdc[17870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:21.658092 17883 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:49:21.659691 17883 webserver.cc:460] Webserver started at http://127.14.153.62:37417/ using document root <none> and password file <none>
I20260619 16:49:21.660468 17883 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:21.660573 17883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:21.660868 17883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:21.664067 17883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-qb36"
server_key: "49831b76e10e444b326f952e1da5c8ab"
server_key_iv: "cb45e7d71d069206c82581827efb43ae"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.664831 17883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-qb36"
server_key: "49831b76e10e444b326f952e1da5c8ab"
server_key_iv: "cb45e7d71d069206c82581827efb43ae"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.669971 17883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:49:21.673590 17898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.675138 17883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 16:49:21.675288 17883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-qb36"
server_key: "49831b76e10e444b326f952e1da5c8ab"
server_key_iv: "cb45e7d71d069206c82581827efb43ae"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.675454 17883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:21.701658 17883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:21.706203 17883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:21.706449 17883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:21.724782 17883 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:34135
I20260619 16:49:21.724810 17949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:34135 every 8 connection(s)
I20260619 16:49:21.726182 17883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:21.729878 17950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:21.733422 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 17883
I20260619 16:49:21.733661 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:21.734468 14948 external_mini_cluster.cc:1428] Setting key 63a9315ccb246e611845bf04378fe281
I20260619 16:49:21.743081 17950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05: Bootstrap starting.
I20260619 16:49:21.747008 17950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:21.748404 17950 log.cc:824] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:49:21 dist-test-slave-qb36 krb5kdc[17870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:21.751536 17950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05: No bootstrap required, opened a new log
I20260619 16:49:21.758116 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:21.736362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.14.153.62:34135)
0619 16:49:21.737129 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:21.737142 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:21.737184 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:49:21.738799 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:21.738823 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:21.739341 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:21.739698 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:21.741603 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.742937 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.744750 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.745469 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.745603 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:21.750570 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:21.750578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:21.750594 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:21.750632 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:21.753776 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:21.754561 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:21.754568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:21.754575 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:21.754749 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:21.755192 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:21.755196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:21.755198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:21.755800 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:21.756135 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:21.756601 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 16:49:21.757028 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":353,"threads_started":1}
I20260619 16:49:21.759603 17950 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34135 } }
I20260619 16:49:21.760011 17950 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:21.760116 17950 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0e13f8cd1e4ff8af7d457feca9ca05, State: Initialized, Role: FOLLOWER
I20260619 16:49:21.760767 17950 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34135 } }
I20260619 16:49:21.761023 17950 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:21.761314 17950 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:21.761505 17950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:21.762194 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:21.763891 17950 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34135 } }
I20260619 16:49:21.764374 17950 leader_election.cc:302] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd0e13f8cd1e4ff8af7d457feca9ca05; no voters:
I20260619 16:49:21.764954 17950 leader_election.cc:288] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:21.765107 17955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:21.766911 17950 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:21.770119 17955 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [term 1 LEADER]: Becoming Leader. State: Replica: bd0e13f8cd1e4ff8af7d457feca9ca05, State: Running, Role: LEADER
I20260619 16:49:21.770658 17955 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34135 } }
I20260619 16:49:21.775600 17956 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34135 } } }
I20260619 16:49:21.776079 17956 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:21.776696 17957 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0e13f8cd1e4ff8af7d457feca9ca05. Latest consensus state: current_term: 1 leader_uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e13f8cd1e4ff8af7d457feca9ca05" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34135 } } }
I20260619 16:49:21.777019 17957 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:21.780689 17967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:21.791473 17967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:21.801927 17967 catalog_manager.cc:1347] Generated new cluster ID: 9292e4bd663c48c19f8e18a36bdf10f6
I20260619 16:49:21.802066 17967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:21.814893 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:21.805493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.14.153.62:34135)
0619 16:49:21.805669 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:21.805676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:21.806284 (+ 608us) server_negotiation.cc:407] Connection header received
0619 16:49:21.806562 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:21.806571 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:21.806672 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:21.806786 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:21.807995 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.809170 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.811012 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.811754 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.811901 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:21.813105 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:21.813111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:21.813117 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:21.814057 (+ 940us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:21.814134 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:21.814170 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:21.814283 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:21.814414 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 16:49:21.814511 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:49:21.818297 17967 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:21.819725 17967 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:21.830199 17967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd0e13f8cd1e4ff8af7d457feca9ca05: Generated new TSK 0
I20260619 16:49:21.831148 17967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:21.841476 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17883
I20260619 16:49:21.842738 17945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:21.932365 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 17883
2026-06-19T16:49:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1920 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:22 dist-test-slave-qb36 krb5kdc[17985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:22 dist-test-slave-qb36 krb5kdc[17985](info): set up 2 sockets
Jun 19 16:49:22 dist-test-slave-qb36 krb5kdc[17985](info): commencing operation
krb5kdc: starting...
W20260619 16:49:23.024317 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:23 dist-test-slave-qb36 krb5kdc[17985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:49:23.219295 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:41105
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:41105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:23.540988 17998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:23.541330 17998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:23.541496 17998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:23.548791 17998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:23.548939 17998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:23.549038 17998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:23.549136 17998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:23.549216 17998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:23.559218 17998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:41105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:41105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:23.560599 17998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:23.561928 17998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:23.573648 18006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:23.573671 18004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:23.573887 18003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:23.574424 17998 server_base.cc:1034] running on GCE node
I20260619 16:49:23.575397 17998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:23.576843 17998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:23.578078 17998 hybrid_clock.cc:648] HybridClock initialized: now 1781887763578008 us; error 54 us; skew 500 ppm
I20260619 16:49:23.578459 17998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:23 dist-test-slave-qb36 krb5kdc[17985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:23.581966 17998 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:49:23.583449 17998 webserver.cc:460] Webserver started at http://127.14.153.62:39809/ using document root <none> and password file <none>
I20260619 16:49:23.584092 17998 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:23.584192 17998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:23.584434 17998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:23.587878 17998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "bfba1559e1c546dd93b5608da09749dc"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-qb36"
server_key: "e3a6996b879bdf03dfadc28428c51f21"
server_key_iv: "b50f1063e89238aa88c7a837f35ad4a7"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.588632 17998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "bfba1559e1c546dd93b5608da09749dc"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-qb36"
server_key: "e3a6996b879bdf03dfadc28428c51f21"
server_key_iv: "b50f1063e89238aa88c7a837f35ad4a7"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.593928 17998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:49:23.597558 18013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:23.598973 17998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:23.599422 17998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "bfba1559e1c546dd93b5608da09749dc"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-qb36"
server_key: "e3a6996b879bdf03dfadc28428c51f21"
server_key_iv: "b50f1063e89238aa88c7a837f35ad4a7"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.599623 17998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:23.628687 17998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:23.632993 17998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:23.633306 17998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:23.650837 17998 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:41105
I20260619 16:49:23.650867 18064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:41105 every 8 connection(s)
I20260619 16:49:23.652309 17998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:23.656107 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:23.657792 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 17998
I20260619 16:49:23.658011 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:23.658663 14948 external_mini_cluster.cc:1428] Setting key c98cb341adb1f529f587e8ae02ef350b
I20260619 16:49:23.669848 18065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc: Bootstrap starting.
I20260619 16:49:23.673336 18065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:23.674556 18065 log.cc:824] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:49:23 dist-test-slave-qb36 krb5kdc[17985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:23.677556 18065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc: No bootstrap required, opened a new log
I20260619 16:49:23.682904 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:23.660712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.14.153.62:41105)
0619 16:49:23.661670 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:23.661685 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:23.661739 (+ 54us) server_negotiation.cc:407] Connection header received
0619 16:49:23.664100 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:23.664130 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:23.664662 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:23.665123 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:23.666879 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.668999 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.670337 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.671345 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.671562 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:23.675728 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:23.675736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:23.675755 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:23.675790 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:23.678871 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:23.679605 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:23.679612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:23.679618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:23.679793 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:23.680178 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:23.680184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:23.680187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:23.680721 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:23.681066 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:23.681475 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 16:49:23.681902 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":438,"threads_started":1}
I20260619 16:49:23.685774 18065 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfba1559e1c546dd93b5608da09749dc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41105 } }
I20260619 16:49:23.686300 18065 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:23.686406 18065 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfba1559e1c546dd93b5608da09749dc, State: Initialized, Role: FOLLOWER
I20260619 16:49:23.686967 18065 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfba1559e1c546dd93b5608da09749dc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41105 } }
I20260619 16:49:23.687254 18065 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:23.687389 18065 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:23.687556 18065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:23.690028 18065 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfba1559e1c546dd93b5608da09749dc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41105 } }
I20260619 16:49:23.690582 18065 leader_election.cc:302] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfba1559e1c546dd93b5608da09749dc; no voters:
I20260619 16:49:23.691340 18065 leader_election.cc:288] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:23.691453 18070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:23.692274 18070 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [term 1 LEADER]: Becoming Leader. State: Replica: bfba1559e1c546dd93b5608da09749dc, State: Running, Role: LEADER
I20260619 16:49:23.692812 18070 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfba1559e1c546dd93b5608da09749dc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41105 } }
I20260619 16:49:23.693545 18065 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:23.696157 18072 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfba1559e1c546dd93b5608da09749dc. Latest consensus state: current_term: 1 leader_uuid: "bfba1559e1c546dd93b5608da09749dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfba1559e1c546dd93b5608da09749dc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41105 } } }
I20260619 16:49:23.696255 18071 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfba1559e1c546dd93b5608da09749dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfba1559e1c546dd93b5608da09749dc" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41105 } } }
I20260619 16:49:23.696542 18072 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:23.696651 18071 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc [sys.catalog]: This master's current role is: LEADER
W20260619 16:49:23.706449 18086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:23.706631 18086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:23.707306 18078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:23.708990 18078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:23.717279 18078 catalog_manager.cc:1347] Generated new cluster ID: 55217d965be044aeb8d5f15c6a0dfeeb
I20260619 16:49:23.717401 18078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:23.726770 18078 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:23.728263 18078 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:23.735011 18078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfba1559e1c546dd93b5608da09749dc: Generated new TSK 0
I20260619 16:49:23.735880 18078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:49:23.788471 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41105
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:24.120446 18091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:24.120731 18091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:24.120913 18091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:24.128244 18091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:24.128357 18091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:24.128532 18091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:24.139292 18091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:24.140640 18091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:24.141960 18091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:24.155303 18099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.155304 18097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.155580 18096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:24.156000 18091 server_base.cc:1034] running on GCE node
I20260619 16:49:24.156601 18091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:24.157516 18091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:24.158720 18091 hybrid_clock.cc:648] HybridClock initialized: now 1781887764158668 us; error 49 us; skew 500 ppm
I20260619 16:49:24.159209 18091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:24 dist-test-slave-qb36 krb5kdc[17985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887764, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:24.162590 18091 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:49:24.164083 18091 webserver.cc:460] Webserver started at http://127.14.153.1:46181/ using document root <none> and password file <none>
I20260619 16:49:24.164790 18091 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:24.164942 18091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:24.165239 18091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:24.168409 18091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "2b8d7d0b516b438f8d7bb684ebee6f7c"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-qb36"
server_key: "45d3c7e653125d1c79e50f6909d2ba41"
server_key_iv: "1703f8c741653fda48854e18acd76790"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.169236 18091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "2b8d7d0b516b438f8d7bb684ebee6f7c"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-qb36"
server_key: "45d3c7e653125d1c79e50f6909d2ba41"
server_key_iv: "1703f8c741653fda48854e18acd76790"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.174474 18091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:49:24.178205 18106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.179616 18091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:24.179792 18091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "2b8d7d0b516b438f8d7bb684ebee6f7c"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-qb36"
server_key: "45d3c7e653125d1c79e50f6909d2ba41"
server_key_iv: "1703f8c741653fda48854e18acd76790"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.179994 18091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:24.200344 18091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:24.204941 18091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:24.205277 18091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:24.206315 18091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:24.208096 18091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:24.208190 18091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.208317 18091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:24.208390 18091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.242097 18091 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:34117
I20260619 16:49:24.242156 18218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:34117 every 8 connection(s)
I20260619 16:49:24.243474 18091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:24.252233 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 18091
I20260619 16:49:24.252454 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:24.253072 14948 external_mini_cluster.cc:1428] Setting key 6ff9edcc7938773653cf254323f8906b
Jun 19 16:49:24 dist-test-slave-qb36 krb5kdc[17985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887764, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:24.264251 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.246075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:50387 (local address 127.14.153.62:41105)
0619 16:49:24.246491 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.246496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.247632 (+ 1136us) server_negotiation.cc:407] Connection header received
0619 16:49:24.249423 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.249428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.249506 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.249613 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.251202 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.251934 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.253669 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.254132 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.254254 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.257747 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.257756 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.257764 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:24.257809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.260968 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:24.261904 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:24.261908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:24.261910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:24.262038 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:24.262496 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:24.262499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:24.262501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:24.262745 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.262849 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.263807 (+ 958us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.263969 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":203,"threads_started":1}
I20260619 16:49:24.265797 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.246703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41105 (local address 127.14.153.1:50387)
0619 16:49:24.247458 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.247510 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.249112 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.249824 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.249833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.250381 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:24.250965 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.250984 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.252129 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.252133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.253512 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.253535 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.254335 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.254340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.254564 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.255406 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:24.255434 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:24.257531 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:24.261105 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:24.261116 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:24.261137 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:24.261789 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:24.262139 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:24.262147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:24.262154 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:24.262390 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:24.262869 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:24.262880 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:24.263394 (+ 514us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.263710 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.264305 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":313,"threads_started":1}
I20260619 16:49:24.267428 18219 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41105
I20260619 16:49:24.267782 18219 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:24.268522 18219 heartbeater.cc:511] Master 127.14.153.62:41105 requested a full tablet report, sending...
I20260619 16:49:24.270514 18030 ts_manager.cc:194] Registered new tserver with Master: 2b8d7d0b516b438f8d7bb684ebee6f7c (127.14.153.1:34117)
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:24.272058 18030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:50387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:49:24.298928 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41105
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:24.618765 18225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:24.619103 18225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:24.619205 18225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:24.626526 18225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:24.626641 18225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:24.626755 18225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:24.636555 18225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:24.637770 18225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:24.639118 18225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:24.651774 18231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.652359 18230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.652765 18233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:24.653070 18225 server_base.cc:1034] running on GCE node
I20260619 16:49:24.653743 18225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:24.654847 18225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:24.656080 18225 hybrid_clock.cc:648] HybridClock initialized: now 1781887764656012 us; error 61 us; skew 500 ppm
I20260619 16:49:24.656450 18225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:24 dist-test-slave-qb36 krb5kdc[17985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887764, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:24.660069 18225 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:49:24.661574 18225 webserver.cc:460] Webserver started at http://127.14.153.2:40141/ using document root <none> and password file <none>
I20260619 16:49:24.662257 18225 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:24.662403 18225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:24.662701 18225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:24.665621 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "41cf0e88d57241e6874fb9035f679628"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-qb36"
server_key: "ae7dda38d4a68e22b591049ab93f7be9"
server_key_iv: "f2aae097c4724a28c8f01474256da47a"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.666460 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "41cf0e88d57241e6874fb9035f679628"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-qb36"
server_key: "ae7dda38d4a68e22b591049ab93f7be9"
server_key_iv: "f2aae097c4724a28c8f01474256da47a"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.671702 18225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260619 16:49:24.675174 18240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.676617 18225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:49:24.676784 18225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "41cf0e88d57241e6874fb9035f679628"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-qb36"
server_key: "ae7dda38d4a68e22b591049ab93f7be9"
server_key_iv: "f2aae097c4724a28c8f01474256da47a"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.676977 18225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:24.717911 18225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:24.722352 18225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:24.722676 18225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:24.723693 18225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:24.725035 18225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:24.725132 18225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.725278 18225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:24.725349 18225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.757458 18225 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:38825
I20260619 16:49:24.757498 18352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:38825 every 8 connection(s)
I20260619 16:49:24.758790 18225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:24.767755 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 18225
I20260619 16:49:24.767936 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:24.768465 14948 external_mini_cluster.cc:1428] Setting key 8457f012fe8ca4089fbb2eb0931551c3
Jun 19 16:49:24 dist-test-slave-qb36 krb5kdc[17985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887764, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:24.780068 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.761387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:56985 (local address 127.14.153.62:41105)
0619 16:49:24.761550 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.761554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.762643 (+ 1089us) server_negotiation.cc:407] Connection header received
0619 16:49:24.764461 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.764465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.764533 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.764629 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.766089 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.766787 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.768569 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.769062 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.769199 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.773167 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.773176 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.773183 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:24.773231 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.776444 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:24.777428 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:24.777433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:24.777435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:24.777561 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:24.778099 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:24.778103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:24.778106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:24.778408 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.778521 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.779469 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.779717 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:49:24.781694 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.761830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41105 (local address 127.14.153.2:56985)
0619 16:49:24.762483 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.762525 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.764196 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.764827 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.764835 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.765367 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:24.765921 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.765939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.766919 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.766922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.768423 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.768441 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.769340 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.769348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.769620 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.770347 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:24.770383 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:24.772933 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:24.776582 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:24.776593 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:24.776613 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:24.777261 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:24.777668 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:24.777676 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:24.777683 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:24.777938 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:24.778539 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:24.778550 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:24.779002 (+ 452us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.779333 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.779996 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":252,"threads_started":1}
I20260619 16:49:24.783216 18353 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41105
I20260619 16:49:24.783574 18353 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:24.784395 18353 heartbeater.cc:511] Master 127.14.153.62:41105 requested a full tablet report, sending...
I20260619 16:49:24.785997 18030 ts_manager.cc:194] Registered new tserver with Master: 41cf0e88d57241e6874fb9035f679628 (127.14.153.2:38825)
I20260619 16:49:24.786784 18030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:56985
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:49:24.814486 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41105
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:25.135726 18358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:25.136047 18358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:25.136201 18358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:25.143479 18358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:25.143618 18358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:25.143792 18358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:25.153738 18358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:25.155185 18358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:25.156534 18358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:25.169409 18364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:25.169498 18366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:25.169883 18363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:25.170343 18358 server_base.cc:1034] running on GCE node
I20260619 16:49:25.170992 18358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:25.172075 18358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:25.173264 18358 hybrid_clock.cc:648] HybridClock initialized: now 1781887765173219 us; error 37 us; skew 500 ppm
I20260619 16:49:25.173581 18358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[17985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887765, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:25.176957 18358 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:49:25.178460 18358 webserver.cc:460] Webserver started at http://127.14.153.3:42105/ using document root <none> and password file <none>
I20260619 16:49:25.179229 18358 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:25.179374 18358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:25.179675 18358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:25.182757 18358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "a78db7af132e473cad08acd3c1930ef0"
format_stamp: "Formatted at 2026-06-19 16:49:25 on dist-test-slave-qb36"
server_key: "92b419914da301f714ca2fe0a85d0980"
server_key_iv: "b68a3fc2416f598dd5abfeb8df2ecf09"
server_key_version: "encryptionkey@0"
I20260619 16:49:25.183666 18358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "a78db7af132e473cad08acd3c1930ef0"
format_stamp: "Formatted at 2026-06-19 16:49:25 on dist-test-slave-qb36"
server_key: "92b419914da301f714ca2fe0a85d0980"
server_key_iv: "b68a3fc2416f598dd5abfeb8df2ecf09"
server_key_version: "encryptionkey@0"
I20260619 16:49:25.189061 18358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:25.192633 18373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:25.194037 18358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:49:25.194204 18358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "a78db7af132e473cad08acd3c1930ef0"
format_stamp: "Formatted at 2026-06-19 16:49:25 on dist-test-slave-qb36"
server_key: "92b419914da301f714ca2fe0a85d0980"
server_key_iv: "b68a3fc2416f598dd5abfeb8df2ecf09"
server_key_version: "encryptionkey@0"
I20260619 16:49:25.194433 18358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:25.210194 18358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:25.214463 18358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:25.214753 18358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:25.215792 18358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:25.217092 18358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:25.217280 18358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:25.217468 18358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:25.217542 18358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:25.249262 18485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:44553 every 8 connection(s)
I20260619 16:49:25.249260 18358 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:44553
I20260619 16:49:25.250558 18358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:25.253302 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 18358
I20260619 16:49:25.253479 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:25.254052 14948 external_mini_cluster.cc:1428] Setting key b89e33bb67892bdd3ee005ca827723aa
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[17985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887765, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:25.271063 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.253027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:48119 (local address 127.14.153.62:41105)
0619 16:49:25.253201 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.253206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.254719 (+ 1513us) server_negotiation.cc:407] Connection header received
0619 16:49:25.257161 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.257166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.257290 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.257397 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.259150 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.259892 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.261395 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.261786 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.261878 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.265315 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.265320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.265323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:25.265355 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.267981 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.268830 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.268834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.268836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.269031 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.269573 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.269577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.269580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.269884 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.270002 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.270612 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.270760 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:49:25.272050 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.253692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41105 (local address 127.14.153.3:48119)
0619 16:49:25.254527 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:25.254563 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:25.256831 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:25.257619 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:25.257628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:25.258358 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:25.258940 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:25.258970 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.260043 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.260046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:25.261264 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:25.261275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.261996 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.262001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:25.262208 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.262892 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:25.262915 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:25.265125 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:25.268125 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:25.268134 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:25.268150 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:25.268692 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:25.269148 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:25.269154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:25.269157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:25.269391 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:25.269994 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:25.269999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:25.270307 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 16:49:25.270532 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 16:49:25.270898 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":358,"threads_started":1}
I20260619 16:49:25.273305 18486 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41105
I20260619 16:49:25.273602 18486 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:25.274308 18486 heartbeater.cc:511] Master 127.14.153.62:41105 requested a full tablet report, sending...
I20260619 16:49:25.275241 18219 heartbeater.cc:503] Master 127.14.153.62:41105 was elected leader, sending a full tablet report...
I20260619 16:49:25.275768 18030 ts_manager.cc:194] Registered new tserver with Master: a78db7af132e473cad08acd3c1930ef0 (127.14.153.3:44553)
I20260619 16:49:25.276671 18030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:48119
I20260619 16:49:25.282004 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[17985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887765, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[17985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887765, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:25.329704 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.316061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.14.153.62:41105)
0619 16:49:25.316233 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.316238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.316737 (+ 499us) server_negotiation.cc:407] Connection header received
0619 16:49:25.316891 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.316895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.316962 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.317058 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.318434 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.319281 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.320531 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.320979 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.321080 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.325024 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.325029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.325032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:25.325067 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.327562 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.328165 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.328168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.328170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.328294 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.328617 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.328620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.328622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.328908 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.329087 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.329311 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.329474 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 16:49:25.330329 18030 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42030
I20260619 16:49:25.332950 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18091
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[17985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887765, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:25.348390 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.333874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.14.153.1:34117)
0619 16:49:25.334505 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.334512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.334534 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:49:25.334587 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.334593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.334815 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.334979 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.336150 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.337129 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.338344 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.338899 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.339036 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.342951 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.342963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.342977 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:25.343012 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.345680 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.346320 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.346330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.346333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.346460 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.346819 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.346824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.346827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.347220 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.347523 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.347784 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.348036 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":285,"threads_started":1}
W20260619 16:49:25.349622 18214 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56142
W20260619 16:49:25.350740 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18091: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:49:25.350924 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 18091
I20260619 16:49:25.366494 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18225
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[17985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887765, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:49:25.382632 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.367469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.14.153.2:38825)
0619 16:49:25.367996 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.368002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.368028 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:49:25.368086 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.368092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.368298 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.368475 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.369705 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.370714 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.372088 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.372635 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.372739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.376878 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.376921 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.376946 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:25.376980 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.380085 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.380701 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.380708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.380711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.380823 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.381190 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.381193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.381195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.381545 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.381769 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.382052 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.382243 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":279,"threads_started":1}
W20260619 16:49:25.383953 18348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43108
W20260619 16:49:25.385020 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18225: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:49:25.385156 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 18225
I20260619 16:49:25.401054 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18358
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[17985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887765, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:49:25.417280 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.402046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.14.153.3:44553)
0619 16:49:25.402666 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.402673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.402701 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:49:25.402758 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.402764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.403013 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.403228 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.404549 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.405577 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.406862 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.407485 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.407616 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.411813 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.411823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.411838 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:25.411870 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.414551 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.415322 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.415330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.415332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.415458 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:25.415768 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:25.415771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:25.415773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:25.416134 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.416363 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.416682 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.416870 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":292,"threads_started":1}
W20260619 16:49:25.418633 18481 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57032
W20260619 16:49:25.419798 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:49:25.419953 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 18358
I20260619 16:49:25.434847 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 17998
I20260619 16:49:25.436098 18060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:25.537228 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 17998
2026-06-19T16:49:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3611 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[18503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[18503](info): set up 2 sockets
Jun 19 16:49:25 dist-test-slave-qb36 krb5kdc[18503](info): commencing operation
krb5kdc: starting...
W20260619 16:49:26.728245 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:26 dist-test-slave-qb36 krb5kdc[18503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:49:26.923236 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:40025
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:40025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:27.247247 18516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:27.247562 18516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:27.247743 18516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:27.255812 18516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:27.255951 18516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:27.256031 18516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:27.256116 18516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:27.256196 18516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:27.266429 18516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:40025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:27.267910 18516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:27.269259 18516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:27.281780 18521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:27.281737 18524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:27.281695 18522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:27.282742 18516 server_base.cc:1034] running on GCE node
I20260619 16:49:27.283716 18516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:27.285199 18516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:27.286435 18516 hybrid_clock.cc:648] HybridClock initialized: now 1781887767286370 us; error 47 us; skew 500 ppm
I20260619 16:49:27.286840 18516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:27 dist-test-slave-qb36 krb5kdc[18503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887767, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:27.293422 18516 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:49:27.295138 18516 webserver.cc:460] Webserver started at http://127.14.153.62:39917/ using document root <none> and password file <none>
I20260619 16:49:27.295876 18516 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:27.296023 18516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:27.296334 18516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:27.299383 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "300acc1f5a8c475b8a23ca7fb7d51773"
format_stamp: "Formatted at 2026-06-19 16:49:27 on dist-test-slave-qb36"
server_key: "ae64818204108d01c60e7f945d593999"
server_key_iv: "568fb54f7b1765b254cf04d8a1dec475"
server_key_version: "encryptionkey@0"
I20260619 16:49:27.300168 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "300acc1f5a8c475b8a23ca7fb7d51773"
format_stamp: "Formatted at 2026-06-19 16:49:27 on dist-test-slave-qb36"
server_key: "ae64818204108d01c60e7f945d593999"
server_key_iv: "568fb54f7b1765b254cf04d8a1dec475"
server_key_version: "encryptionkey@0"
I20260619 16:49:27.305274 18516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 16:49:27.308727 18531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:27.310081 18516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:49:27.310259 18516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "300acc1f5a8c475b8a23ca7fb7d51773"
format_stamp: "Formatted at 2026-06-19 16:49:27 on dist-test-slave-qb36"
server_key: "ae64818204108d01c60e7f945d593999"
server_key_iv: "568fb54f7b1765b254cf04d8a1dec475"
server_key_version: "encryptionkey@0"
I20260619 16:49:27.310449 18516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:27.324414 18516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:27.328517 18516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:27.328794 18516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:27.346833 18516 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:40025
I20260619 16:49:27.346872 18582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:40025 every 8 connection(s)
I20260619 16:49:27.348197 18516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:27.351781 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 18516
I20260619 16:49:27.351822 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:27.352005 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:27.352643 14948 external_mini_cluster.cc:1428] Setting key 844eaba82e3aa72bec2455be777313b3
I20260619 16:49:27.366058 18583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773: Bootstrap starting.
Jun 19 16:49:27 dist-test-slave-qb36 krb5kdc[18503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:27.369616 18583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:27.370695 18583 log.cc:824] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:27.374068 18583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773: No bootstrap required, opened a new log
I20260619 16:49:27.377719 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:27.354613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.14.153.62:40025)
0619 16:49:27.355493 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:27.355510 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:27.355561 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:49:27.358379 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:27.358419 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:27.358954 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:27.359456 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:27.360768 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:27.362798 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:27.364107 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:27.365122 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:27.365295 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:27.369572 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:27.369580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:27.369594 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:27.369632 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:27.373245 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:27.373913 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:27.373921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:27.373928 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:27.374150 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:27.374533 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:27.374544 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:27.374549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:27.375264 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:27.375578 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:27.375859 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:49:27.376328 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":384,"threads_started":1}
I20260619 16:49:27.383306 18583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300acc1f5a8c475b8a23ca7fb7d51773" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40025 } }
I20260619 16:49:27.383670 18583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:27.383747 18583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300acc1f5a8c475b8a23ca7fb7d51773, State: Initialized, Role: FOLLOWER
I20260619 16:49:27.384377 18583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300acc1f5a8c475b8a23ca7fb7d51773" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40025 } }
I20260619 16:49:27.384701 18583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:27.384838 18583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:27.385031 18583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:27.387247 18583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300acc1f5a8c475b8a23ca7fb7d51773" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40025 } }
I20260619 16:49:27.387776 18583 leader_election.cc:302] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 300acc1f5a8c475b8a23ca7fb7d51773; no voters:
I20260619 16:49:27.388461 18583 leader_election.cc:288] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:27.388616 18588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:27.389320 18588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [term 1 LEADER]: Becoming Leader. State: Replica: 300acc1f5a8c475b8a23ca7fb7d51773, State: Running, Role: LEADER
I20260619 16:49:27.389863 18588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300acc1f5a8c475b8a23ca7fb7d51773" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40025 } }
I20260619 16:49:27.390532 18583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:27.392865 18589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "300acc1f5a8c475b8a23ca7fb7d51773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300acc1f5a8c475b8a23ca7fb7d51773" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40025 } } }
I20260619 16:49:27.393206 18589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:27.393646 18590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 300acc1f5a8c475b8a23ca7fb7d51773. Latest consensus state: current_term: 1 leader_uuid: "300acc1f5a8c475b8a23ca7fb7d51773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300acc1f5a8c475b8a23ca7fb7d51773" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40025 } } }
I20260619 16:49:27.393949 18590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:27.400447 18596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:27.404991 18596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:27.413467 18596 catalog_manager.cc:1347] Generated new cluster ID: e2138cc3a2d842e384663d7599cace0d
I20260619 16:49:27.413589 18596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:27.455542 18596 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:27.456847 18596 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:27.471915 18596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 300acc1f5a8c475b8a23ca7fb7d51773: Generated new TSK 0
I20260619 16:49:27.472812 18596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:49:27.519577 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40025
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:27.850665 18609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:27.851114 18609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:27.851306 18609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:27.858847 18609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:27.859009 18609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:27.859210 18609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:27.869174 18609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:27.870517 18609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:27.871863 18609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:27.885352 18614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:27.885277 18615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:27.885309 18617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:27.886490 18609 server_base.cc:1034] running on GCE node
I20260619 16:49:27.887214 18609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:27.888154 18609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:27.889371 18609 hybrid_clock.cc:648] HybridClock initialized: now 1781887767889309 us; error 59 us; skew 500 ppm
I20260619 16:49:27.889686 18609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:27 dist-test-slave-qb36 krb5kdc[18503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887767, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:27.892963 18609 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:49:27.894565 18609 webserver.cc:460] Webserver started at http://127.14.153.1:39709/ using document root <none> and password file <none>
I20260619 16:49:27.895525 18609 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:27.895674 18609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:27.896006 18609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:27.898938 18609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "70eb04598085446b899a46f426cfad81"
format_stamp: "Formatted at 2026-06-19 16:49:27 on dist-test-slave-qb36"
server_key: "c716dfc079eb24e7836f77888e0811d0"
server_key_iv: "00ddc8fa13e7f732c60a60b2b521860b"
server_key_version: "encryptionkey@0"
I20260619 16:49:27.899765 18609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "70eb04598085446b899a46f426cfad81"
format_stamp: "Formatted at 2026-06-19 16:49:27 on dist-test-slave-qb36"
server_key: "c716dfc079eb24e7836f77888e0811d0"
server_key_iv: "00ddc8fa13e7f732c60a60b2b521860b"
server_key_version: "encryptionkey@0"
I20260619 16:49:27.904991 18609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:49:27.908388 18624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:27.909735 18609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:27.909900 18609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "70eb04598085446b899a46f426cfad81"
format_stamp: "Formatted at 2026-06-19 16:49:27 on dist-test-slave-qb36"
server_key: "c716dfc079eb24e7836f77888e0811d0"
server_key_iv: "00ddc8fa13e7f732c60a60b2b521860b"
server_key_version: "encryptionkey@0"
I20260619 16:49:27.910102 18609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:27.922749 18609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:27.926548 18609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:27.926829 18609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:27.927790 18609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:27.929045 18609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:27.929143 18609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:27.929284 18609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:27.929358 18609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:27.961730 18609 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:44857
I20260619 16:49:27.961777 18736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:44857 every 8 connection(s)
I20260619 16:49:27.963068 18609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:27.970441 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 18609
I20260619 16:49:27.970665 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:27.971334 14948 external_mini_cluster.cc:1428] Setting key ed3cf5ea53c10ecda9455da2a4223bfa
Jun 19 16:49:27 dist-test-slave-qb36 krb5kdc[18503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887767, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:27.984601 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:27.965555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:51507 (local address 127.14.153.62:40025)
0619 16:49:27.965947 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:27.965952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:27.967011 (+ 1059us) server_negotiation.cc:407] Connection header received
0619 16:49:27.968806 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:27.968812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:27.968902 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:27.969023 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:27.970444 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:27.971485 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:27.973443 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:27.973898 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:27.974059 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:27.978191 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:27.978196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:27.978199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:27.978229 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:27.981416 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:27.982102 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:27.982111 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:27.982117 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:27.982269 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:27.982672 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:27.982679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:27.982685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:27.983103 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:27.983248 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:27.983889 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 16:49:27.984144 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":191,"threads_started":1}
I20260619 16:49:27.985261 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:27.966163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40025 (local address 127.14.153.1:51507)
0619 16:49:27.966854 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:27.966890 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:27.968549 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:27.969247 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:27.969255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:27.969756 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:27.970287 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:27.970303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:27.971646 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:27.971652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:27.973300 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:27.973312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:27.974128 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:27.974137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:27.974486 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:27.975300 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:27.975336 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:27.977967 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:27.981551 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:27.981558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:27.981572 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:27.981982 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:27.982387 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:27.982390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:27.982393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:27.982575 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:27.983211 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:27.983217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:27.983534 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 16:49:27.983757 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 16:49:27.984172 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":277,"threads_started":1}
I20260619 16:49:27.986789 18737 heartbeater.cc:347] Connected to a master server at 127.14.153.62:40025
I20260619 16:49:27.987190 18737 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:27.988179 18737 heartbeater.cc:511] Master 127.14.153.62:40025 requested a full tablet report, sending...
I20260619 16:49:27.990636 18548 ts_manager.cc:194] Registered new tserver with Master: 70eb04598085446b899a46f426cfad81 (127.14.153.1:44857)
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:27.992281 18548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:51507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:49:28.018112 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40025
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:28.336380 18743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:28.336651 18743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:28.336808 18743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:28.343917 18743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:28.344024 18743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:28.344219 18743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:28.353938 18743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:28.355295 18743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:28.356603 18743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:28.369783 18751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.369863 18748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.369874 18749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:28.370652 18743 server_base.cc:1034] running on GCE node
I20260619 16:49:28.371331 18743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:28.372287 18743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:28.373494 18743 hybrid_clock.cc:648] HybridClock initialized: now 1781887768373447 us; error 37 us; skew 500 ppm
I20260619 16:49:28.373814 18743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:28 dist-test-slave-qb36 krb5kdc[18503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.376880 18743 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:49:28.378309 18743 webserver.cc:460] Webserver started at http://127.14.153.2:41679/ using document root <none> and password file <none>
I20260619 16:49:28.379005 18743 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:28.379195 18743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:28.379506 18743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:28.382378 18743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "1009ad03971f41b0a4bb34e0e9677cc0"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-qb36"
server_key: "b5676397a10bf1ea5bda152b2f7d379a"
server_key_iv: "9b374f005b90d15780fb543fa800a494"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.383535 18743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "1009ad03971f41b0a4bb34e0e9677cc0"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-qb36"
server_key: "b5676397a10bf1ea5bda152b2f7d379a"
server_key_iv: "9b374f005b90d15780fb543fa800a494"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.388764 18743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:28.392350 18758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.393679 18743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:28.393843 18743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "1009ad03971f41b0a4bb34e0e9677cc0"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-qb36"
server_key: "b5676397a10bf1ea5bda152b2f7d379a"
server_key_iv: "9b374f005b90d15780fb543fa800a494"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.394063 18743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:28.434283 18743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:28.438678 18743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:28.439006 18743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:28.440066 18743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:28.441430 18743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:28.441537 18743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.441684 18743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:28.441752 18743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.474112 18743 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:45069
I20260619 16:49:28.474197 18870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:45069 every 8 connection(s)
I20260619 16:49:28.475409 18743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:28.477131 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 18743
I20260619 16:49:28.477341 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:28.477919 14948 external_mini_cluster.cc:1428] Setting key 9f4d49bd8b21dbc071f03f0105571db0
Jun 19 16:49:28 dist-test-slave-qb36 krb5kdc[18503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:28.497442 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.479287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:60209 (local address 127.14.153.62:40025)
0619 16:49:28.479450 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.479455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.479684 (+ 229us) server_negotiation.cc:407] Connection header received
0619 16:49:28.481693 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.481702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.481793 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.481919 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.483437 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.484609 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.486180 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.486596 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.486733 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.490227 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:28.490236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:28.490243 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:28.490289 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:28.493458 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.494518 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.494522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.494525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.494661 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.495236 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.495240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.495242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.495697 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:28.495831 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:28.496868 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0619 16:49:28.497146 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:49:28.498490 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.478777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40025 (local address 127.14.153.2:60209)
0619 16:49:28.479551 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:28.479596 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:28.481414 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:28.482121 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:28.482129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:28.482621 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:28.483267 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.483286 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.484766 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.484770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.486032 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.486043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.486826 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.486833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.487129 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.487813 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:28.487833 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:28.489983 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:28.493628 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.493641 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.493666 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.494366 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.494778 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.494785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.494790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.495095 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.495849 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:28.495866 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:28.496372 (+ 506us) client_negotiation.cc:769] Sending connection context
0619 16:49:28.496711 (+ 339us) client_negotiation.cc:240] Negotiation successful
0619 16:49:28.497216 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
I20260619 16:49:28.500057 18871 heartbeater.cc:347] Connected to a master server at 127.14.153.62:40025
I20260619 16:49:28.500394 18871 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:28.501312 18871 heartbeater.cc:511] Master 127.14.153.62:40025 requested a full tablet report, sending...
I20260619 16:49:28.502914 18548 ts_manager.cc:194] Registered new tserver with Master: 1009ad03971f41b0a4bb34e0e9677cc0 (127.14.153.2:45069)
I20260619 16:49:28.503772 18548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:60209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:49:28.522181 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40025
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:28.846683 18876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:28.846982 18876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:28.847158 18876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:28.854338 18876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:28.854453 18876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:28.854682 18876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:28.864423 18876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:28.865759 18876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:28.867081 18876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:28.880060 18882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.880198 18884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.880136 18881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:28.881225 18876 server_base.cc:1034] running on GCE node
I20260619 16:49:28.881914 18876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:28.882896 18876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:28.884092 18876 hybrid_clock.cc:648] HybridClock initialized: now 1781887768884044 us; error 44 us; skew 500 ppm
I20260619 16:49:28.884426 18876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:28 dist-test-slave-qb36 krb5kdc[18503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.887621 18876 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:49:28.889127 18876 webserver.cc:460] Webserver started at http://127.14.153.3:33585/ using document root <none> and password file <none>
I20260619 16:49:28.889887 18876 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:28.890002 18876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:28.890264 18876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:28.893440 18876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "1d8184a6401342d692191f90644c762d"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-qb36"
server_key: "faa3bcb6b5bede9cb861560fcb1a28a0"
server_key_iv: "e651c1e3a1e3b6dff921033e06bab331"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.894158 18876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "1d8184a6401342d692191f90644c762d"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-qb36"
server_key: "faa3bcb6b5bede9cb861560fcb1a28a0"
server_key_iv: "e651c1e3a1e3b6dff921033e06bab331"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.899747 18876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:49:28.903257 18891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.904651 18876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:28.904824 18876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "1d8184a6401342d692191f90644c762d"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-qb36"
server_key: "faa3bcb6b5bede9cb861560fcb1a28a0"
server_key_iv: "e651c1e3a1e3b6dff921033e06bab331"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.905031 18876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:28.926191 18876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:28.930374 18876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:28.930655 18876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:28.931734 18876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:28.933017 18876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:28.933113 18876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.933251 18876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:28.933320 18876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.969179 18876 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:38399
I20260619 16:49:28.969231 19003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:38399 every 8 connection(s)
I20260619 16:49:28.970394 18876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:28.970922 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 18876
I20260619 16:49:28.971128 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:28.971697 14948 external_mini_cluster.cc:1428] Setting key d089969c9f94f4b6924b7c25e130028a
Jun 19 16:49:28 dist-test-slave-qb36 krb5kdc[18503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:28.990566 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.973107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:39029 (local address 127.14.153.62:40025)
0619 16:49:28.973265 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.973269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.974696 (+ 1427us) server_negotiation.cc:407] Connection header received
0619 16:49:28.976591 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.976595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.976658 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.976740 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.978310 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.979144 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.980624 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.981056 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.981152 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.984543 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:28.984548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:28.984551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:28.984583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:28.987757 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.988482 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.988486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.988488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.988613 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.989012 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.989017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.989019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.989304 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:28.989437 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:28.990115 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 16:49:28.990293 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:49:28.991609 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.973744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40025 (local address 127.14.153.3:39029)
0619 16:49:28.974548 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:28.974599 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:28.976338 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:28.977063 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:28.977074 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:28.977596 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:28.978156 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.978173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.979257 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.979261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.980500 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.980511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.981234 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.981238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.981425 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.982128 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:28.982154 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:28.984323 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:28.987894 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.987902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.987921 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.988363 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.988707 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.988710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.988712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.988893 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.989421 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:28.989428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:28.989749 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 16:49:28.989986 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 16:49:28.990376 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":340,"threads_started":1}
I20260619 16:49:28.992959 19004 heartbeater.cc:347] Connected to a master server at 127.14.153.62:40025
I20260619 16:49:28.993225 19004 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:28.993945 19004 heartbeater.cc:511] Master 127.14.153.62:40025 requested a full tablet report, sending...
I20260619 16:49:28.995306 18548 ts_manager.cc:194] Registered new tserver with Master: 1d8184a6401342d692191f90644c762d (127.14.153.3:38399)
I20260619 16:49:28.995445 18737 heartbeater.cc:503] Master 127.14.153.62:40025 was elected leader, sending a full tablet report...
I20260619 16:49:28.996402 18548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:39029
I20260619 16:49:28.999507 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:49:29 dist-test-slave-qb36 krb5kdc[18503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:49:29 dist-test-slave-qb36 krb5kdc[18503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:29.047995 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:29.033041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.14.153.1:44857)
0619 16:49:29.033720 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:29.033727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:29.033753 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:49:29.033812 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:29.033819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:29.034038 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:29.034206 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:29.035720 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.036742 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.037967 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.038556 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.038698 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:29.042563 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:29.042575 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:29.042590 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:29.042627 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:29.045223 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.045888 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.045895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.045898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.046021 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.046390 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.046395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.046402 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.046757 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:29.047088 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:29.047364 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 16:49:29.047644 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":344,"threads_started":1}
Jun 19 16:49:29 dist-test-slave-qb36 krb5kdc[18503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:29.065526 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:29.052046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.14.153.62:40025)
0619 16:49:29.052265 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:29.052275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:29.052635 (+ 360us) server_negotiation.cc:407] Connection header received
0619 16:49:29.052863 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:29.052872 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:29.053000 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:29.053089 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:29.054265 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.054962 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.056218 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.056645 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.056750 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:29.060555 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:29.060560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:29.060564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:29.060598 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:29.063290 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.063876 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.063880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.063882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.064025 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.064311 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.064314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.064316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.064602 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:29.064770 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:29.065029 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 16:49:29.065194 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 16:49:29.066102 18548 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43984
I20260619 16:49:29.067102 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18609
I20260619 16:49:29.080161 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:29.067914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.14.153.1:44857)
0619 16:49:29.068128 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:29.068135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:29.068330 (+ 195us) server_negotiation.cc:407] Connection header received
0619 16:49:29.068441 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:29.068444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:29.068504 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:29.068581 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:29.069836 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.070632 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.071899 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.072305 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.072427 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:29.074952 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:29.074957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:29.074960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:29.074991 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:29.077619 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.078252 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.078256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.078259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.078390 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.078725 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.078729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.078731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.079015 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:29.079319 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:29.079639 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 16:49:29.079803 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:49:29.080788 18732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:29.166579 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 18609
I20260619 16:49:29.184051 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18743
Jun 19 16:49:29 dist-test-slave-qb36 krb5kdc[18503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:49:29.201283 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:29.185074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.14.153.2:45069)
0619 16:49:29.185619 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:29.185626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:29.185653 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:49:29.185715 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:29.185721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:29.185988 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:29.186143 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:29.187558 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.188601 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.189877 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.190542 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.190643 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:29.195159 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:29.195174 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:29.195192 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:29.195229 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:29.198126 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.199001 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.199008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.199011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.199231 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.199619 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.199622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.199624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.200033 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:29.200245 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:29.200639 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 16:49:29.200839 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":277,"threads_started":1}
I20260619 16:49:29.202647 18866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:29.286808 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 18743
I20260619 16:49:29.302815 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18876
Jun 19 16:49:29 dist-test-slave-qb36 krb5kdc[18503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:49:29.319375 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:29.303973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.14.153.3:38399)
0619 16:49:29.304606 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:29.304614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:29.304642 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:49:29.304701 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:29.304707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:29.304920 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:29.305049 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:29.306293 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.307414 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.308657 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:29.309353 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:29.309463 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:29.313645 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:29.313656 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:29.313671 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:29.313708 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:29.316677 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.317373 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.317379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.317382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.317500 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:29.317892 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:29.317897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:29.317899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:29.318254 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:29.318498 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:29.318718 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 16:49:29.318921 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":346,"threads_started":1}
I20260619 16:49:29.320674 18999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:29.404692 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 18876
I20260619 16:49:29.420353 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 18516
I20260619 16:49:29.421497 18578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:29.523391 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 18516
2026-06-19T16:49:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3987 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:29 dist-test-slave-qb36 krb5kdc[19023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:29 dist-test-slave-qb36 krb5kdc[19023](info): set up 2 sockets
Jun 19 16:49:29 dist-test-slave-qb36 krb5kdc[19023](info): commencing operation
krb5kdc: starting...
W20260619 16:49:30.644332 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:30 dist-test-slave-qb36 krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:49:30.846428 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:38227
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:31.168699 19036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:31.169013 19036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:31.169178 19036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:31.176395 19036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:31.176506 19036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:31.176553 19036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:31.176599 19036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:31.176646 19036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:31.186312 19036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:38227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:31.187678 19036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:31.189016 19036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:31.200477 19042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.200575 19044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.200555 19041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.201445 19036 server_base.cc:1034] running on GCE node
I20260619 16:49:31.202219 19036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:31.203753 19036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:31.204990 19036 hybrid_clock.cc:648] HybridClock initialized: now 1781887771204941 us; error 62 us; skew 500 ppm
I20260619 16:49:31.205371 19036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:31 dist-test-slave-qb36 krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:31.208484 19036 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:49:31.209712 19036 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.209843 19036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.210188 19036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:31.213224 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "e8267e466a954ea1b07a21cbcf0494d5"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-qb36"
server_key: "3e782cf52fecc5f1aacd6d4e92616af5"
server_key_iv: "8c311d529a587a0297ddaba8c82b1bc1"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.214036 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "e8267e466a954ea1b07a21cbcf0494d5"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-qb36"
server_key: "3e782cf52fecc5f1aacd6d4e92616af5"
server_key_iv: "8c311d529a587a0297ddaba8c82b1bc1"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.219923 19036 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 16:49:31.223462 19050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.224774 19036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:49:31.224964 19036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "e8267e466a954ea1b07a21cbcf0494d5"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-qb36"
server_key: "3e782cf52fecc5f1aacd6d4e92616af5"
server_key_iv: "8c311d529a587a0297ddaba8c82b1bc1"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.225201 19036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.237864 19036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:31.241809 19036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:31.242096 19036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:31.260080 19036 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:38227
I20260619 16:49:31.260097 19101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:38227 every 8 connection(s)
I20260619 16:49:31.261029 19036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:31.264958 19102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:31.265046 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 19036
I20260619 16:49:31.265256 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:31.266045 14948 external_mini_cluster.cc:1428] Setting key 145206df05c6efdb80e74764b84b40df
I20260619 16:49:31.279155 19102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5: Bootstrap starting.
Jun 19 16:49:31 dist-test-slave-qb36 krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:31.282866 19102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:31.284140 19102 log.cc:824] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:31.287196 19102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5: No bootstrap required, opened a new log
I20260619 16:49:31.290194 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.268181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.14.153.62:38227)
0619 16:49:31.269077 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.269091 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.269133 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:49:31.271425 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.271447 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.271874 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.272217 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.274055 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.275420 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.277233 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.277897 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.278033 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.282646 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.282654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.282669 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.282706 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.285848 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.286592 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.286605 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.286612 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.286773 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.287243 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.287250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.287253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.287848 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.288187 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.288680 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.289116 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":431,"threads_started":1}
I20260619 16:49:31.295280 19102 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8267e466a954ea1b07a21cbcf0494d5" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38227 } }
I20260619 16:49:31.295670 19102 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:31.295786 19102 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8267e466a954ea1b07a21cbcf0494d5, State: Initialized, Role: FOLLOWER
I20260619 16:49:31.296347 19102 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8267e466a954ea1b07a21cbcf0494d5" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38227 } }
I20260619 16:49:31.296571 19102 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:31.296705 19102 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:31.296865 19102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:31.299437 19102 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8267e466a954ea1b07a21cbcf0494d5" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38227 } }
I20260619 16:49:31.299994 19102 leader_election.cc:302] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8267e466a954ea1b07a21cbcf0494d5; no voters:
I20260619 16:49:31.300493 19102 leader_election.cc:288] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:31.300647 19107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:31.301307 19107 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [term 1 LEADER]: Becoming Leader. State: Replica: e8267e466a954ea1b07a21cbcf0494d5, State: Running, Role: LEADER
I20260619 16:49:31.301898 19107 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8267e466a954ea1b07a21cbcf0494d5" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38227 } }
I20260619 16:49:31.302591 19102 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:31.305099 19109 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8267e466a954ea1b07a21cbcf0494d5. Latest consensus state: current_term: 1 leader_uuid: "e8267e466a954ea1b07a21cbcf0494d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8267e466a954ea1b07a21cbcf0494d5" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38227 } } }
I20260619 16:49:31.305486 19109 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:31.305150 19108 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8267e466a954ea1b07a21cbcf0494d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8267e466a954ea1b07a21cbcf0494d5" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38227 } } }
I20260619 16:49:31.305828 19108 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:31.309095 19113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:31.314682 19113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:31.324141 19113 catalog_manager.cc:1347] Generated new cluster ID: 59c047e923bd46dd944354ca7b49fd7f
I20260619 16:49:31.324404 19113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:31.356105 19113 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:31.357509 19113 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:31.367941 19113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5: Generated new TSK 0
I20260619 16:49:31.368830 19113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:49:31.418702 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:38227
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:31.747238 19128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:31.747514 19128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:31.747610 19128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:31.754566 19128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:31.754678 19128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:31.754793 19128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:31.764385 19128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:38227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:31.765733 19128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:31.767102 19128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:31.779801 19133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.780053 19134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.780331 19136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.783207 19128 server_base.cc:1034] running on GCE node
I20260619 16:49:31.784053 19128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:31.785427 19128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:31.786677 19128 hybrid_clock.cc:648] HybridClock initialized: now 1781887771786606 us; error 66 us; skew 500 ppm
I20260619 16:49:31.787114 19128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:31 dist-test-slave-qb36 krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:31.791138 19128 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:49:31.792289 19128 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.792382 19128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.792635 19128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:31.795585 19128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "f9414022521b46548375f99a37f771ac"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-qb36"
server_key: "ccbb5e33e2ea42fb9c430aff95231859"
server_key_iv: "d144678a9ef203cdd052482a79dbb988"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.796392 19128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "f9414022521b46548375f99a37f771ac"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-qb36"
server_key: "ccbb5e33e2ea42fb9c430aff95231859"
server_key_iv: "d144678a9ef203cdd052482a79dbb988"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.801397 19128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:49:31.805076 19142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.806497 19128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:49:31.806649 19128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "f9414022521b46548375f99a37f771ac"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-qb36"
server_key: "ccbb5e33e2ea42fb9c430aff95231859"
server_key_iv: "d144678a9ef203cdd052482a79dbb988"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.806855 19128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.851246 19128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:31.855695 19128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:31.855999 19128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:31.856997 19128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:31.858378 19128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:31.858462 19128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.858620 19128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:31.858714 19128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.891886 19128 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:45743
I20260619 16:49:31.891999 19254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:45743 every 8 connection(s)
I20260619 16:49:31.892758 19128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:31.899446 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 19128
I20260619 16:49:31.899668 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:31.900455 14948 external_mini_cluster.cc:1428] Setting key e6917419c8c068d1b66920d5bf093273
Jun 19 16:49:31 dist-test-slave-qb36 krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:31.914963 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.895243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:41543 (local address 127.14.153.62:38227)
0619 16:49:31.895692 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.895697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.896569 (+ 872us) server_negotiation.cc:407] Connection header received
0619 16:49:31.898421 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.898427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.898517 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.898632 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.900740 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.901577 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.903589 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.904392 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.904543 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.908580 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.908585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.908589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.908621 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.911661 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.912627 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.912631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.912634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.912748 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.913223 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.913226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.913228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.913479 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.913585 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.914530 (+ 945us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.914699 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":253,"threads_started":1}
I20260619 16:49:31.916627 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.895702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:38227 (local address 127.14.153.1:41543)
0619 16:49:31.896402 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.896447 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.898137 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.898888 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.898897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.899716 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.900555 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.900581 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.901755 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.901761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.903448 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.903460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.904641 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.904647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.904871 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.905626 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.905654 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.908282 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.911802 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.911835 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.911857 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.912492 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.912860 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.912868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.912875 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.913112 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.913605 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.913616 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.914099 (+ 483us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.914414 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.915021 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":274,"threads_started":1}
I20260619 16:49:31.918322 19255 heartbeater.cc:347] Connected to a master server at 127.14.153.62:38227
I20260619 16:49:31.918637 19255 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:31.919374 19255 heartbeater.cc:511] Master 127.14.153.62:38227 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:31.921522 19067 ts_manager.cc:194] Registered new tserver with Master: f9414022521b46548375f99a37f771ac (127.14.153.1:45743)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
I20260619 16:49:31.923115 19067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:41543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:49:31.946861 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:38227
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:32.272658 19261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:32.273034 19261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:32.273180 19261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:32.284765 19261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:32.284886 19261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:32.285005 19261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:32.295341 19261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:38227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:32.296629 19261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:32.297950 19261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:32.311700 19266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:32.311774 19267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:32.312150 19269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:32.312901 19261 server_base.cc:1034] running on GCE node
I20260619 16:49:32.313546 19261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:32.314534 19261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:32.315796 19261 hybrid_clock.cc:648] HybridClock initialized: now 1781887772315719 us; error 64 us; skew 500 ppm
I20260619 16:49:32.316229 19261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:32 dist-test-slave-qb36 krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887772, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:32.323340 19261 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:49:32.324609 19261 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:32.324745 19261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:32.325119 19261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:32.328253 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "75680d0e5ac2438ea5c4279e529a624b"
format_stamp: "Formatted at 2026-06-19 16:49:32 on dist-test-slave-qb36"
server_key: "afac171b3fdd59e955ed32edd8aedcc9"
server_key_iv: "68bf93788c866a51baba8a703c9f74db"
server_key_version: "encryptionkey@0"
I20260619 16:49:32.329092 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "75680d0e5ac2438ea5c4279e529a624b"
format_stamp: "Formatted at 2026-06-19 16:49:32 on dist-test-slave-qb36"
server_key: "afac171b3fdd59e955ed32edd8aedcc9"
server_key_iv: "68bf93788c866a51baba8a703c9f74db"
server_key_version: "encryptionkey@0"
I20260619 16:49:32.334558 19261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 16:49:32.338459 19275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:32.340075 19261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:32.340256 19261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "75680d0e5ac2438ea5c4279e529a624b"
format_stamp: "Formatted at 2026-06-19 16:49:32 on dist-test-slave-qb36"
server_key: "afac171b3fdd59e955ed32edd8aedcc9"
server_key_iv: "68bf93788c866a51baba8a703c9f74db"
server_key_version: "encryptionkey@0"
I20260619 16:49:32.340528 19261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:32.372612 19261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:32.377236 19261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:32.377524 19261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:32.378507 19261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:32.379930 19261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:32.380014 19261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:32.380138 19261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:32.380215 19261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:32.413894 19261 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:42775
I20260619 16:49:32.413942 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:42775 every 8 connection(s)
I20260619 16:49:32.414798 19261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:32.416828 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 19261
I20260619 16:49:32.417021 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:32.417649 14948 external_mini_cluster.cc:1428] Setting key 85863d3115f773c37fc718c7f284f6e3
Jun 19 16:49:32 dist-test-slave-qb36 krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887772, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:32.438194 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.417397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:40369 (local address 127.14.153.62:38227)
0619 16:49:32.419394 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.419399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.419418 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:49:32.421524 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.421533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.421625 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.421748 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.423477 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.424621 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.426327 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.427154 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.427296 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.430957 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.430966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.430975 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.431018 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.434767 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.435621 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.435630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.435635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.435792 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.436201 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.436206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.436210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.436598 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.436759 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.437512 (+ 753us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.437773 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1886,"thread_start_us":1794,"threads_started":1}
I20260619 16:49:32.438994 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.418067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:38227 (local address 127.14.153.2:40369)
0619 16:49:32.419215 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.419261 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.421239 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.422022 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.422032 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.422557 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:32.423290 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.423314 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.424782 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.424785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.426165 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.426178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.427369 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.427374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.427564 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.428291 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:32.428320 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:32.430708 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:32.434934 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.434943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.434958 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.435494 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.435907 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.435910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.435912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.436081 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.436760 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:32.436767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:32.437117 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.437382 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.437772 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":659,"threads_started":1}
I20260619 16:49:32.440616 19388 heartbeater.cc:347] Connected to a master server at 127.14.153.62:38227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
I20260619 16:49:32.440999 19388 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:32.441668 19388 heartbeater.cc:511] Master 127.14.153.62:38227 requested a full tablet report, sending...
I20260619 16:49:32.443214 19067 ts_manager.cc:194] Registered new tserver with Master: 75680d0e5ac2438ea5c4279e529a624b (127.14.153.2:42775)
I20260619 16:49:32.444105 19067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:40369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:49:32.466122 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:38227
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:42019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:32.787873 19394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:32.788167 19394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:32.788264 19394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:32.795552 19394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:32.795663 19394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:32.795781 19394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:32.805644 19394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:42019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:38227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:32.806891 19394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:32.808220 19394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:32.820580 19400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:32.820818 19399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:32.821341 19402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:32.821976 19394 server_base.cc:1034] running on GCE node
I20260619 16:49:32.822587 19394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:32.823621 19394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:32.824867 19394 hybrid_clock.cc:648] HybridClock initialized: now 1781887772824794 us; error 50 us; skew 500 ppm
I20260619 16:49:32.825184 19394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:32 dist-test-slave-qb36 krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887772, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:32.828234 19394 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:49:32.829488 19394 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:32.829622 19394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:32.829967 19394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:32.833043 19394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "da5f52da307042ab946ff332856ee49c"
format_stamp: "Formatted at 2026-06-19 16:49:32 on dist-test-slave-qb36"
server_key: "e399abb1b8138985d360ca79d058bfd0"
server_key_iv: "c9ebeb5e9710d29a6aa9ecf7bddc3775"
server_key_version: "encryptionkey@0"
I20260619 16:49:32.833834 19394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "da5f52da307042ab946ff332856ee49c"
format_stamp: "Formatted at 2026-06-19 16:49:32 on dist-test-slave-qb36"
server_key: "e399abb1b8138985d360ca79d058bfd0"
server_key_iv: "c9ebeb5e9710d29a6aa9ecf7bddc3775"
server_key_version: "encryptionkey@0"
I20260619 16:49:32.839088 19394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 16:49:32.842597 19408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:32.844236 19394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:49:32.844419 19394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "da5f52da307042ab946ff332856ee49c"
format_stamp: "Formatted at 2026-06-19 16:49:32 on dist-test-slave-qb36"
server_key: "e399abb1b8138985d360ca79d058bfd0"
server_key_iv: "c9ebeb5e9710d29a6aa9ecf7bddc3775"
server_key_version: "encryptionkey@0"
I20260619 16:49:32.844652 19394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:32.872653 19394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:32.876942 19394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:32.877228 19394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:32.878238 19394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:32.879866 19394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:32.879983 19394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:32.880095 19394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:32.880177 19394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:32.914139 19394 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:34919
I20260619 16:49:32.914224 19520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:34919 every 8 connection(s)
I20260619 16:49:32.915011 19394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:32.925192 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 19394
I20260619 16:49:32.925414 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:32.925945 14948 external_mini_cluster.cc:1428] Setting key c9b3819b9239a3aff94ae053fa7295fa
I20260619 16:49:32.926159 19255 heartbeater.cc:503] Master 127.14.153.62:38227 was elected leader, sending a full tablet report...
Jun 19 16:49:32 dist-test-slave-qb36 krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887772, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:32.935200 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.917485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:34175 (local address 127.14.153.62:38227)
0619 16:49:32.917697 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.917702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.918881 (+ 1179us) server_negotiation.cc:407] Connection header received
0619 16:49:32.920730 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.920735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.920819 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.920919 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.922587 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.923500 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.925070 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.925691 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.925895 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.929982 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.930006 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.930014 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.930052 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.932562 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.933245 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.933249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.933251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.933367 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.933717 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.933720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.933722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.933996 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.934109 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.934711 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.934908 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:49:32.936199 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.918025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:38227 (local address 127.14.153.3:34175)
0619 16:49:32.918708 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.918752 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.920455 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.921234 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.921243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.921782 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:32.922383 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.922401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.923637 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.923641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.924907 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.924926 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.926068 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.926077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.926361 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.927128 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:32.927163 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:32.929795 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:32.932682 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.932688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.932700 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.933125 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.933466 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.933468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.933471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.933630 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.934104 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:32.934110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:32.934406 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.934637 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.935011 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":280,"threads_started":1}
I20260619 16:49:32.937493 19521 heartbeater.cc:347] Connected to a master server at 127.14.153.62:38227
I20260619 16:49:32.937794 19521 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:32.938406 19521 heartbeater.cc:511] Master 127.14.153.62:38227 requested a full tablet report, sending...
I20260619 16:49:32.939882 19066 ts_manager.cc:194] Registered new tserver with Master: da5f52da307042ab946ff332856ee49c (127.14.153.3:34919)
I20260619 16:49:32.941098 19066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:34175
I20260619 16:49:32.941422 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:32.958577 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.946522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.14.153.62:38227)
0619 16:49:32.946727 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.946736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.947196 (+ 460us) server_negotiation.cc:407] Connection header received
0619 16:49:32.947338 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.947341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.947420 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.947498 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.949063 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.949779 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.950946 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.951423 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.951522 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.953628 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.953633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.953636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.953664 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.956093 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.956713 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.956718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.956720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.956862 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.957186 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.957189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.957191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.957481 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.957671 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.958044 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.958198 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:49:32.962734 19066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:32.969477 19066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:33.007970 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.982244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45743 (local address 127.0.0.1:34804)
0619 16:49:32.999372 (+ 17128us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.999387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.999496 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:33.000384 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:33.000395 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:33.000431 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:33.000994 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.001013 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.002383 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.002391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.004887 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.004903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.007149 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.007154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.007329 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.007362 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:49:33.007437 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:49:33.007540 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16993,"mutex_wait_us":1,"thread_start_us":15098,"threads_started":1}
I20260619 16:49:33.008214 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.982521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.14.153.3:34919)
0619 16:49:32.999831 (+ 17310us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.999844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.999888 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:49:33.000008 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:33.000019 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:33.000244 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:33.000767 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:33.001336 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.003054 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.005634 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.007002 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.007192 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.007416 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:33.007676 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 16:49:33.007835 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17116,"thread_start_us":16946,"threads_started":1}
I20260619 16:49:33.008488 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.981564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.14.153.2:42775)
0619 16:49:33.000396 (+ 18832us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:33.000405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:33.000431 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:49:33.000520 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:33.000530 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:33.000694 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:33.001672 (+ 978us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:33.001904 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.003361 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.005485 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.006604 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.006768 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.006967 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:33.007949 (+ 982us) server_negotiation.cc:299] Negotiation successful
0619 16:49:33.008157 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18667,"thread_start_us":18529,"threads_started":1}
I20260619 16:49:33.008667 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.981738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:34919 (local address 127.0.0.1:44224)
0619 16:49:32.998669 (+ 16931us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.998694 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.998839 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:33.000471 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:33.000477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:33.000505 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:33.001089 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.001106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.003199 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.003204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.005454 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.005467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.007262 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.007269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.007458 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.007491 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:49:33.007621 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 16:49:33.007766 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16770,"mutex_wait_us":64,"spinlock_wait_cycles":6912,"thread_start_us":16531,"threads_started":1}
I20260619 16:49:33.008100 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.981380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:42775 (local address 127.0.0.1:55730)
0619 16:49:32.996825 (+ 15445us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.996896 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.997121 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:33.000953 (+ 3832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:33.000965 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:33.000992 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:33.001461 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.001472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.003499 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.003509 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.005301 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.005320 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.006797 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.006803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.006994 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.007082 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 16:49:33.007232 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 16:49:33.007455 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15082,"mutex_wait_us":1,"thread_start_us":17704,"threads_started":1}
I20260619 16:49:33.009270 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.983015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.14.153.1:45743)
0619 16:49:32.993276 (+ 10261us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.993290 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.999640 (+ 6350us) server_negotiation.cc:407] Connection header received
0619 16:49:32.999756 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.999766 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:33.000095 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:33.000307 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:33.001189 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.002236 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.005091 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.006596 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.006732 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.006984 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:33.008226 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0619 16:49:33.008405 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10029,"thread_start_us":9884,"threads_started":1}
I20260619 16:49:33.012651 19323 tablet_service.cc:1467] Processing CreateTablet for tablet 18c9e76e39444d5f9d01500b0bff9e95 (DEFAULT_TABLE table=test-table [id=ec6f3fa59eda4b17807881b82ffb9f8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:33.012651 19190 tablet_service.cc:1467] Processing CreateTablet for tablet 18c9e76e39444d5f9d01500b0bff9e95 (DEFAULT_TABLE table=test-table [id=ec6f3fa59eda4b17807881b82ffb9f8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:33.012653 19456 tablet_service.cc:1467] Processing CreateTablet for tablet 18c9e76e39444d5f9d01500b0bff9e95 (DEFAULT_TABLE table=test-table [id=ec6f3fa59eda4b17807881b82ffb9f8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:33.013935 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c9e76e39444d5f9d01500b0bff9e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:33.014003 19323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c9e76e39444d5f9d01500b0bff9e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:33.014003 19190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c9e76e39444d5f9d01500b0bff9e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:33.025095 19539 tablet_bootstrap.cc:492] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: Bootstrap starting.
I20260619 16:49:33.028158 19539 tablet_bootstrap.cc:654] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:33.028875 19540 tablet_bootstrap.cc:492] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: Bootstrap starting.
I20260619 16:49:33.029598 19539 log.cc:824] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:33.030848 19541 tablet_bootstrap.cc:492] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: Bootstrap starting.
I20260619 16:49:33.033002 19540 tablet_bootstrap.cc:654] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:33.034044 19541 tablet_bootstrap.cc:654] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:33.034664 19540 log.cc:824] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:33.035094 19541 log.cc:824] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:33.037976 19541 tablet_bootstrap.cc:492] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: No bootstrap required, opened a new log
I20260619 16:49:33.038129 19539 tablet_bootstrap.cc:492] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: No bootstrap required, opened a new log
I20260619 16:49:33.038129 19540 tablet_bootstrap.cc:492] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: No bootstrap required, opened a new log
I20260619 16:49:33.038311 19541 ts_tablet_manager.cc:1397] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 16:49:33.038400 19539 ts_tablet_manager.cc:1397] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260619 16:49:33.038400 19540 ts_tablet_manager.cc:1397] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 16:49:33.045900 19541 raft_consensus.cc:348] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } }
I20260619 16:49:33.046375 19541 raft_consensus.cc:374] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:33.046473 19541 raft_consensus.cc:729] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75680d0e5ac2438ea5c4279e529a624b, State: Initialized, Role: FOLLOWER
I20260619 16:49:33.047158 19541 consensus_queue.cc:260] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } }
I20260619 16:49:33.048260 19388 heartbeater.cc:503] Master 127.14.153.62:38227 was elected leader, sending a full tablet report...
I20260619 16:49:33.049260 19541 ts_tablet_manager.cc:1428] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 16:49:33.049157 19540 raft_consensus.cc:348] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } }
I20260619 16:49:33.049190 19539 raft_consensus.cc:348] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } }
I20260619 16:49:33.049618 19540 raft_consensus.cc:374] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:33.049616 19539 raft_consensus.cc:374] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:33.049713 19540 raft_consensus.cc:729] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9414022521b46548375f99a37f771ac, State: Initialized, Role: FOLLOWER
I20260619 16:49:33.049713 19539 raft_consensus.cc:729] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5f52da307042ab946ff332856ee49c, State: Initialized, Role: FOLLOWER
I20260619 16:49:33.050271 19539 consensus_queue.cc:260] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } }
I20260619 16:49:33.050289 19540 consensus_queue.cc:260] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } }
I20260619 16:49:33.052050 19539 ts_tablet_manager.cc:1428] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260619 16:49:33.052416 19540 ts_tablet_manager.cc:1428] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260619 16:49:33.052707 19521 heartbeater.cc:503] Master 127.14.153.62:38227 was elected leader, sending a full tablet report...
I20260619 16:49:33.103503 19547 raft_consensus.cc:482] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:33.103814 19547 raft_consensus.cc:504] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } }
I20260619 16:49:33.105782 19547 leader_election.cc:288] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75680d0e5ac2438ea5c4279e529a624b (127.14.153.2:42775), f9414022521b46548375f99a37f771ac (127.14.153.1:45743)
I20260619 16:49:33.113457 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.106368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:46213 (local address 127.14.153.1:45743)
0619 16:49:33.106566 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:33.106572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:33.106592 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:49:33.106660 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:33.106664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:33.106741 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:33.106842 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:33.107486 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.108503 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.111110 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.112464 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.112635 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.112709 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:33.113078 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 16:49:33.113174 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:49:33.113547 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.106004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45743 (local address 127.14.153.3:46213)
0619 16:49:33.106329 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:33.106354 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:33.106474 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:33.107051 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:33.107056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:33.107088 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:33.107353 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.107361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.108680 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.108684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.110906 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.110921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.112635 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.112639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.112889 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.112932 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 16:49:33.113047 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 16:49:33.113189 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":38,"spinlock_wait_cycles":24448}
I20260619 16:49:33.113626 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.106894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:38831 (local address 127.14.153.2:42775)
0619 16:49:33.107130 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:33.107134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:33.107149 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:49:33.107297 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:33.107301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:33.107355 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:33.107477 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:33.107967 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.108891 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.111498 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.112709 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.112843 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.112895 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:33.113246 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 16:49:33.113371 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:49:33.113708 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.106123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:42775 (local address 127.14.153.3:38831)
0619 16:49:33.107057 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:33.107073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:33.107178 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:33.107512 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:33.107515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:33.107534 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:33.107777 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.107783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.109040 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.109045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.111290 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:33.111302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.112913 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.112919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:33.113101 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.113130 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:49:33.113207 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:49:33.113340 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":33,"thread_start_us":611,"threads_started":1}
I20260619 16:49:33.114835 19343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9e76e39444d5f9d01500b0bff9e95" candidate_uuid: "da5f52da307042ab946ff332856ee49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75680d0e5ac2438ea5c4279e529a624b" is_pre_election: true
I20260619 16:49:33.115067 19210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9e76e39444d5f9d01500b0bff9e95" candidate_uuid: "da5f52da307042ab946ff332856ee49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9414022521b46548375f99a37f771ac" is_pre_election: true
I20260619 16:49:33.115469 19343 raft_consensus.cc:2454] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da5f52da307042ab946ff332856ee49c in term 0.
I20260619 16:49:33.115653 19210 raft_consensus.cc:2454] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da5f52da307042ab946ff332856ee49c in term 0.
I20260619 16:49:33.116170 19411 leader_election.cc:302] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75680d0e5ac2438ea5c4279e529a624b, da5f52da307042ab946ff332856ee49c; no voters:
I20260619 16:49:33.116564 19547 raft_consensus.cc:2781] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:33.116705 19547 raft_consensus.cc:482] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:33.116824 19547 raft_consensus.cc:3037] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:33.119007 19547 raft_consensus.cc:504] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } }
I20260619 16:49:33.119742 19547 leader_election.cc:288] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [CANDIDATE]: Term 1 election: Requested vote from peers 75680d0e5ac2438ea5c4279e529a624b (127.14.153.2:42775), f9414022521b46548375f99a37f771ac (127.14.153.1:45743)
I20260619 16:49:33.120190 19343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9e76e39444d5f9d01500b0bff9e95" candidate_uuid: "da5f52da307042ab946ff332856ee49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75680d0e5ac2438ea5c4279e529a624b"
I20260619 16:49:33.120234 19210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c9e76e39444d5f9d01500b0bff9e95" candidate_uuid: "da5f52da307042ab946ff332856ee49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9414022521b46548375f99a37f771ac"
I20260619 16:49:33.120409 19343 raft_consensus.cc:3037] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:33.120424 19210 raft_consensus.cc:3037] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:33.122565 19210 raft_consensus.cc:2454] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5f52da307042ab946ff332856ee49c in term 1.
I20260619 16:49:33.122699 19343 raft_consensus.cc:2454] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5f52da307042ab946ff332856ee49c in term 1.
I20260619 16:49:33.122979 19412 leader_election.cc:302] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da5f52da307042ab946ff332856ee49c, f9414022521b46548375f99a37f771ac; no voters:
I20260619 16:49:33.123381 19547 raft_consensus.cc:2781] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:33.123762 19547 raft_consensus.cc:686] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 1 LEADER]: Becoming Leader. State: Replica: da5f52da307042ab946ff332856ee49c, State: Running, Role: LEADER
I20260619 16:49:33.124253 19547 consensus_queue.cc:237] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } }
I20260619 16:49:33.129371 19065 catalog_manager.cc:5495] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c reported cstate change: term changed from 0 to 1, leader changed from <none> to da5f52da307042ab946ff332856ee49c (127.14.153.3). New cstate: current_term: 1 leader_uuid: "da5f52da307042ab946ff332856ee49c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da5f52da307042ab946ff332856ee49c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 34919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 } health_report { overall_health: UNKNOWN } } }
W20260619 16:49:33.145045 19256 tablet.cc:2369] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:33.148269 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.142271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.14.153.3:34919)
0619 16:49:33.142472 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:33.142476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:33.142557 (+ 81us) server_negotiation.cc:407] Connection header received
0619 16:49:33.142771 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:33.142775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:33.142886 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:33.143013 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:33.143468 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.144361 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.146028 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.146465 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.146565 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.147091 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:33.147617 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:33.147728 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:33.147910 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 16:49:33.147978 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 16:49:33.156483 19210 raft_consensus.cc:1261] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [term 1 FOLLOWER]: Refusing update from remote peer da5f52da307042ab946ff332856ee49c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:33.156484 19343 raft_consensus.cc:1261] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [term 1 FOLLOWER]: Refusing update from remote peer da5f52da307042ab946ff332856ee49c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:33.157617 19549 consensus_queue.cc:1035] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9414022521b46548375f99a37f771ac" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:33.158022 19547 consensus_queue.cc:1035] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [LEADER]: Connected to new peer: Peer: permanent_uuid: "75680d0e5ac2438ea5c4279e529a624b" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:49:33.166585 19522 tablet.cc:2369] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:33.167379 19389 tablet.cc:2369] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:33.184090 19065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:33.184396 19065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:33.188338 19065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8267e466a954ea1b07a21cbcf0494d5: Sending DeleteTablet for 3 replicas of tablet 18c9e76e39444d5f9d01500b0bff9e95
I20260619 16:49:33.189612 19456 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c9e76e39444d5f9d01500b0bff9e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:33 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:44224
I20260619 16:49:33.189605 19323 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c9e76e39444d5f9d01500b0bff9e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:33 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:55730
I20260619 16:49:33.189702 19190 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c9e76e39444d5f9d01500b0bff9e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:33 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:34804
I20260619 16:49:33.192519 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 19128
I20260619 16:49:33.202098 19564 tablet_replica.cc:330] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: stopping tablet replica
I20260619 16:49:33.202993 19563 tablet_replica.cc:330] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: stopping tablet replica
I20260619 16:49:33.203656 19563 raft_consensus.cc:2229] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:33.203878 19564 raft_consensus.cc:2229] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:33.204459 19564 raft_consensus.cc:2258] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:33.204470 19563 raft_consensus.cc:2258] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:33.207180 19563 ts_tablet_manager.cc:1905] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:33.207763 19562 tablet_replica.cc:330] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: stopping tablet replica
I20260619 16:49:33.207748 19564 ts_tablet_manager.cc:1905] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:33.208283 19562 raft_consensus.cc:2229] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:33.210003 19562 raft_consensus.cc:2258] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 16:49:33 dist-test-slave-qb36 krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:33.215220 19564 ts_tablet_manager.cc:1918] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:33.215428 19564 log.cc:1196] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/18c9e76e39444d5f9d01500b0bff9e95
I20260619 16:49:33.215967 19564 ts_tablet_manager.cc:1939] T 18c9e76e39444d5f9d01500b0bff9e95 P f9414022521b46548375f99a37f771ac: Deleting consensus metadata
I20260619 16:49:33.216301 19562 ts_tablet_manager.cc:1905] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:33.217949 19054 catalog_manager.cc:4841] TS f9414022521b46548375f99a37f771ac (127.14.153.1:45743): tablet 18c9e76e39444d5f9d01500b0bff9e95 (table test-table [id=ec6f3fa59eda4b17807881b82ffb9f8c]) successfully deleted
I20260619 16:49:33.221398 19563 ts_tablet_manager.cc:1918] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:33.221493 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.194088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.14.153.1:45743)
0619 16:49:33.194268 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:33.194274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:33.199166 (+ 4892us) server_negotiation.cc:407] Connection header received
0619 16:49:33.199939 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:33.199945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:33.200037 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:33.200148 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:33.203733 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.204959 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.207115 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.207795 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.207902 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.213598 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:33.213616 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:33.213635 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:33.213688 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:33.217869 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:33.218826 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:33.218835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:33.218839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:33.219017 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:33.219521 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:33.219527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:33.219530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:33.220000 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:33.220291 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:33.220864 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 16:49:33.221090 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:49:33.221577 19563 log.cc:1196] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/18c9e76e39444d5f9d01500b0bff9e95
I20260619 16:49:33.222134 19563 ts_tablet_manager.cc:1939] T 18c9e76e39444d5f9d01500b0bff9e95 P da5f52da307042ab946ff332856ee49c: Deleting consensus metadata
I20260619 16:49:33.222978 19250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:33.224246 19051 catalog_manager.cc:4841] TS da5f52da307042ab946ff332856ee49c (127.14.153.3:34919): tablet 18c9e76e39444d5f9d01500b0bff9e95 (table test-table [id=ec6f3fa59eda4b17807881b82ffb9f8c]) successfully deleted
I20260619 16:49:33.225755 19562 ts_tablet_manager.cc:1918] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:33.225986 19562 log.cc:1196] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/18c9e76e39444d5f9d01500b0bff9e95
I20260619 16:49:33.226650 19562 ts_tablet_manager.cc:1939] T 18c9e76e39444d5f9d01500b0bff9e95 P 75680d0e5ac2438ea5c4279e529a624b: Deleting consensus metadata
I20260619 16:49:33.228945 19053 catalog_manager.cc:4841] TS 75680d0e5ac2438ea5c4279e529a624b (127.14.153.2:42775): tablet 18c9e76e39444d5f9d01500b0bff9e95 (table test-table [id=ec6f3fa59eda4b17807881b82ffb9f8c]) successfully deleted
I20260619 16:49:33.332541 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 19128
I20260619 16:49:33.349778 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 19261
Jun 19 16:49:33 dist-test-slave-qb36 krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:49:33.367197 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.350562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.14.153.2:42775)
0619 16:49:33.350997 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:33.351002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:33.351074 (+ 72us) server_negotiation.cc:407] Connection header received
0619 16:49:33.351137 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:33.351141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:33.351199 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:33.351275 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:33.352861 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.353598 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.355588 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.356037 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.356156 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.361263 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:33.361279 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:33.361296 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:33.361340 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:33.364334 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:33.365072 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:33.365080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:33.365082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:33.365206 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:33.365608 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:33.365619 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:33.365621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:33.365997 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:33.366294 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:33.366603 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 16:49:33.366835 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260619 16:49:33.367869 19383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:33.457356 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 19261
I20260619 16:49:33.480525 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 19394
Jun 19 16:49:33 dist-test-slave-qb36 krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:49:33.502911 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.481445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.14.153.3:34919)
0619 16:49:33.481622 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:33.481627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:33.481703 (+ 76us) server_negotiation.cc:407] Connection header received
0619 16:49:33.481967 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:33.481972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:33.482049 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:33.482142 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:33.483938 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.484748 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.486605 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:33.487140 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:33.487239 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:33.493130 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:33.493149 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:33.493170 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:33.493210 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:33.497172 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:33.498930 (+ 1758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:33.498944 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:33.498948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:33.499247 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:33.499963 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:33.499968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:33.499975 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:33.500529 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:33.500875 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:33.502208 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0619 16:49:33.502461 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:49:33.506623 19516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:33.630434 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 19394
I20260619 16:49:33.650058 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 19036
I20260619 16:49:33.651234 19097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:33.757719 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 19036
2026-06-19T16:49:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4232 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T16:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:33Z Disabled control of system clock
I20260619 16:49:33.824754 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35443
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:49:34.150892 19573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:34.151280 19573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:34.151443 19573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:34.159004 19573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:34.159189 19573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:34.159272 19573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:34.159377 19573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:34.159463 19573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:34.168941 19573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:34.170347 19573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:34.172264 19573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:34.184497 19578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:34.184490 19581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:34.184429 19579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:34.185343 19573 server_base.cc:1034] running on GCE node
I20260619 16:49:34.186177 19573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:34.187695 19573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:34.188898 19573 hybrid_clock.cc:648] HybridClock initialized: now 1781887774188856 us; error 39 us; skew 500 ppm
I20260619 16:49:34.189270 19573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:49:34.191711 19573 webserver.cc:460] Webserver started at http://127.14.153.62:45655/ using document root <none> and password file <none>
I20260619 16:49:34.192413 19573 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:34.192509 19573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:34.192831 19573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:34.195885 19573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "e8ab416dff834c309b617b14379ab958"
format_stamp: "Formatted at 2026-06-19 16:49:34 on dist-test-slave-qb36"
server_key: "525c3e739bd99dc320927efe3d5a7437"
server_key_iv: "a7454d31b38fdceb0070875638e2c8ce"
server_key_version: "encryptionkey@0"
I20260619 16:49:34.196724 19573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "e8ab416dff834c309b617b14379ab958"
format_stamp: "Formatted at 2026-06-19 16:49:34 on dist-test-slave-qb36"
server_key: "525c3e739bd99dc320927efe3d5a7437"
server_key_iv: "a7454d31b38fdceb0070875638e2c8ce"
server_key_version: "encryptionkey@0"
I20260619 16:49:34.202097 19573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:49:34.205600 19587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:34.207120 19573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:34.207324 19573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "e8ab416dff834c309b617b14379ab958"
format_stamp: "Formatted at 2026-06-19 16:49:34 on dist-test-slave-qb36"
server_key: "525c3e739bd99dc320927efe3d5a7437"
server_key_iv: "a7454d31b38fdceb0070875638e2c8ce"
server_key_version: "encryptionkey@0"
I20260619 16:49:34.207552 19573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:34.213385 19573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:34.214161 19573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:34.214459 19573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:34.233165 19573 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35443
I20260619 16:49:34.233213 19638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35443 every 8 connection(s)
I20260619 16:49:34.234597 19573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:34.238339 19639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:34.244112 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 19573
I20260619 16:49:34.244315 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:34.245119 14948 external_mini_cluster.cc:1428] Setting key 78761459b1f3b7e90ab854d417705e1d
I20260619 16:49:34.249967 19639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958: Bootstrap starting.
I20260619 16:49:34.253352 19639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:34.253544 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.247363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.14.153.62:35443)
0619 16:49:34.248393 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:34.248403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:34.248443 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:49:34.250004 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:34.250025 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:34.250475 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:34.250835 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:34.251949 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:34.251956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:34.251977 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:34.251993 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:34.252044 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:34.252062 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:34.252276 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:34.252416 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 16:49:34.252656 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":338,"threads_started":1}
I20260619 16:49:34.254451 19639 log.cc:824] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:34.257251 19639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958: No bootstrap required, opened a new log
I20260619 16:49:34.264717 19639 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ab416dff834c309b617b14379ab958" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35443 } }
I20260619 16:49:34.265092 19639 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:34.265198 19639 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ab416dff834c309b617b14379ab958, State: Initialized, Role: FOLLOWER
I20260619 16:49:34.265847 19639 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ab416dff834c309b617b14379ab958" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35443 } }
I20260619 16:49:34.266083 19639 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:34.266219 19639 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:34.266410 19639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:34.268982 19639 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ab416dff834c309b617b14379ab958" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35443 } }
I20260619 16:49:34.269542 19639 leader_election.cc:302] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8ab416dff834c309b617b14379ab958; no voters:
I20260619 16:49:34.270231 19639 leader_election.cc:288] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:34.270376 19644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:34.271124 19644 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [term 1 LEADER]: Becoming Leader. State: Replica: e8ab416dff834c309b617b14379ab958, State: Running, Role: LEADER
I20260619 16:49:34.271646 19644 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ab416dff834c309b617b14379ab958" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35443 } }
I20260619 16:49:34.272351 19639 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:34.274513 19645 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8ab416dff834c309b617b14379ab958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ab416dff834c309b617b14379ab958" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35443 } } }
I20260619 16:49:34.274744 19646 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8ab416dff834c309b617b14379ab958. Latest consensus state: current_term: 1 leader_uuid: "e8ab416dff834c309b617b14379ab958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ab416dff834c309b617b14379ab958" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35443 } } }
I20260619 16:49:34.274986 19645 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:34.275098 19646 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:34.276783 19653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:34.281966 19653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:34.291345 19653 catalog_manager.cc:1347] Generated new cluster ID: ab42cfb082be4d0e96b66f06dc46aa8a
I20260619 16:49:34.291481 19653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:34.300995 19653 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:34.301442 19653 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:34.307655 19653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958: Generated new TSK 0
I20260619 16:49:34.308480 19653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:34.314549 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35443
--builtin_ntp_servers=127.14.153.20:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:49:34.656937 19663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:34.657215 19663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:34.657369 19663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:34.664636 19663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:34.664739 19663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:34.664918 19663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:34.674171 19663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:34.675637 19663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:34.677544 19663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:34.690500 19671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:34.690732 19668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:34.690505 19669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:34.691424 19663 server_base.cc:1034] running on GCE node
I20260619 16:49:34.691987 19663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:34.692884 19663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:34.694087 19663 hybrid_clock.cc:648] HybridClock initialized: now 1781887774694041 us; error 30 us; skew 500 ppm
I20260619 16:49:34.694511 19663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:49:34.696985 19663 webserver.cc:460] Webserver started at http://127.14.153.1:42707/ using document root <none> and password file <none>
I20260619 16:49:34.697703 19663 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:34.697832 19663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:34.698177 19663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:34.701138 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "345a48ba65dc47539b437adaa52ea987"
format_stamp: "Formatted at 2026-06-19 16:49:34 on dist-test-slave-qb36"
server_key: "1a5be93f7371c3e696220fde0622325a"
server_key_iv: "fab6224809a5c996c8fe388c24547941"
server_key_version: "encryptionkey@0"
I20260619 16:49:34.701936 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "345a48ba65dc47539b437adaa52ea987"
format_stamp: "Formatted at 2026-06-19 16:49:34 on dist-test-slave-qb36"
server_key: "1a5be93f7371c3e696220fde0622325a"
server_key_iv: "fab6224809a5c996c8fe388c24547941"
server_key_version: "encryptionkey@0"
I20260619 16:49:34.707178 19663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 16:49:34.710626 19677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:34.712882 19663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:49:34.713068 19663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "345a48ba65dc47539b437adaa52ea987"
format_stamp: "Formatted at 2026-06-19 16:49:34 on dist-test-slave-qb36"
server_key: "1a5be93f7371c3e696220fde0622325a"
server_key_iv: "fab6224809a5c996c8fe388c24547941"
server_key_version: "encryptionkey@0"
I20260619 16:49:34.713300 19663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:34.719148 19663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:34.719949 19663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:34.720218 19663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:34.721180 19663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:34.722569 19663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:34.722705 19663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:34.722826 19663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:34.722909 19663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:34.755508 19663 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:33777
I20260619 16:49:34.755561 19789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:33777 every 8 connection(s)
I20260619 16:49:34.756897 19663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:34.764948 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.759328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:41279 (local address 127.14.153.62:35443)
0619 16:49:34.759735 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:34.759740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:34.760981 (+ 1241us) server_negotiation.cc:407] Connection header received
0619 16:49:34.762630 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:34.762635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:34.762710 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:34.762828 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:34.764199 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:34.764205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:34.764210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:34.764228 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:34.764275 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:34.764287 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:34.764415 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:34.764603 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 16:49:34.764669 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":207,"threads_started":1}
I20260619 16:49:34.765869 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.759958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35443 (local address 127.14.153.1:41279)
0619 16:49:34.760828 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.760873 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:34.762373 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:34.763155 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:34.763163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:34.763256 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:34.763917 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:34.763919 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:34.763956 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:34.764039 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:34.764390 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:34.764402 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:34.764409 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:49:34.764684 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 16:49:34.764942 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":244,"threads_started":1}
I20260619 16:49:34.766549 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 19663
I20260619 16:49:34.766732 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:34.767465 19790 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35443
I20260619 16:49:34.767550 14948 external_mini_cluster.cc:1428] Setting key 3071c315595be9ccbc0825f42c081870
I20260619 16:49:34.767767 19790 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:34.768463 19790 heartbeater.cc:511] Master 127.14.153.62:35443 requested a full tablet report, sending...
I20260619 16:49:34.770532 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35443
--builtin_ntp_servers=127.14.153.20:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:49:34.770761 19604 ts_manager.cc:194] Registered new tserver with Master: 345a48ba65dc47539b437adaa52ea987 (127.14.153.1:33777)
W20260619 16:49:35.100533 19794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:35.100828 19794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:35.100998 19794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:35.108335 19794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:35.108443 19794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:35.108615 19794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:35.118144 19794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:35.119520 19794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:35.121296 19794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:35.134897 19800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:35.134845 19802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:35.134943 19799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:35.140815 19794 server_base.cc:1034] running on GCE node
I20260619 16:49:35.141507 19794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:35.142525 19794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:35.143747 19794 hybrid_clock.cc:648] HybridClock initialized: now 1781887775143691 us; error 67 us; skew 500 ppm
I20260619 16:49:35.144068 19794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:49:35.146458 19794 webserver.cc:460] Webserver started at http://127.14.153.2:46163/ using document root <none> and password file <none>
I20260619 16:49:35.147284 19794 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:35.147416 19794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:35.147753 19794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:35.150727 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "169c231ada94464b810a09ea006c0c32"
format_stamp: "Formatted at 2026-06-19 16:49:35 on dist-test-slave-qb36"
server_key: "0a48b56dbb03b5db60c24b43b8b1a5b3"
server_key_iv: "3a7ed7f9e46ec59b749b81e593b0e9ff"
server_key_version: "encryptionkey@0"
I20260619 16:49:35.151582 19794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "169c231ada94464b810a09ea006c0c32"
format_stamp: "Formatted at 2026-06-19 16:49:35 on dist-test-slave-qb36"
server_key: "0a48b56dbb03b5db60c24b43b8b1a5b3"
server_key_iv: "3a7ed7f9e46ec59b749b81e593b0e9ff"
server_key_version: "encryptionkey@0"
I20260619 16:49:35.156890 19794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:49:35.160523 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:35.161903 19794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:35.162106 19794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "169c231ada94464b810a09ea006c0c32"
format_stamp: "Formatted at 2026-06-19 16:49:35 on dist-test-slave-qb36"
server_key: "0a48b56dbb03b5db60c24b43b8b1a5b3"
server_key_iv: "3a7ed7f9e46ec59b749b81e593b0e9ff"
server_key_version: "encryptionkey@0"
I20260619 16:49:35.162350 19794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:35.168169 19794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:35.168992 19794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:35.169262 19794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:35.170267 19794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:35.171679 19794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:35.171804 19794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:35.171947 19794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:35.172062 19794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:35.205348 19794 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:39563
I20260619 16:49:35.205404 19920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:39563 every 8 connection(s)
I20260619 16:49:35.206693 19794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:35.210939 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 19794
I20260619 16:49:35.211144 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:35.211721 14948 external_mini_cluster.cc:1428] Setting key 20629f4791299ff14ae86169929b8f99
I20260619 16:49:35.215077 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35443
--builtin_ntp_servers=127.14.153.20:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:49:35.216532 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.209519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:47567 (local address 127.14.153.62:35443)
0619 16:49:35.209719 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.209723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.211543 (+ 1820us) server_negotiation.cc:407] Connection header received
0619 16:49:35.213856 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.213865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.213951 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.214047 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.215633 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.215642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.215650 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.215675 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.215720 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.215737 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.215899 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.216104 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.216175 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:49:35.217449 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.210092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35443 (local address 127.14.153.2:47567)
0619 16:49:35.211313 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.211409 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.213509 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.214338 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.214347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.214454 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:35.215297 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:35.215302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:35.215363 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:35.215460 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:35.215891 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:35.215899 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:35.215905 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.216137 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.216357 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":360,"threads_started":1}
I20260619 16:49:35.218981 19921 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35443
I20260619 16:49:35.219321 19921 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:35.219936 19921 heartbeater.cc:511] Master 127.14.153.62:35443 requested a full tablet report, sending...
I20260619 16:49:35.221489 19604 ts_manager.cc:194] Registered new tserver with Master: 169c231ada94464b810a09ea006c0c32 (127.14.153.2:39563)
W20260619 16:49:35.545377 19924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:35.545671 19924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:35.545758 19924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:35.553170 19924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:35.553279 19924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:35.553387 19924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:35.562588 19924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:35.564028 19924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:35.565819 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:35.579248 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:35.579280 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:35.579344 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:35.580637 19924 server_base.cc:1034] running on GCE node
I20260619 16:49:35.581269 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:35.582183 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:35.583411 19924 hybrid_clock.cc:648] HybridClock initialized: now 1781887775583364 us; error 34 us; skew 500 ppm
I20260619 16:49:35.583711 19924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:49:35.585991 19924 webserver.cc:460] Webserver started at http://127.14.153.3:44711/ using document root <none> and password file <none>
I20260619 16:49:35.586660 19924 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:35.586750 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:35.587152 19924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:35.590922 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "0d9479a6b0eb405d9d68f017b8da3422"
format_stamp: "Formatted at 2026-06-19 16:49:35 on dist-test-slave-qb36"
server_key: "53c7399a7cf9a615640c007348011f1e"
server_key_iv: "c2295155153f56fe17bd1b3ac1ebabf2"
server_key_version: "encryptionkey@0"
I20260619 16:49:35.591811 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "0d9479a6b0eb405d9d68f017b8da3422"
format_stamp: "Formatted at 2026-06-19 16:49:35 on dist-test-slave-qb36"
server_key: "53c7399a7cf9a615640c007348011f1e"
server_key_iv: "c2295155153f56fe17bd1b3ac1ebabf2"
server_key_version: "encryptionkey@0"
I20260619 16:49:35.597071 19924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 16:49:35.600860 19938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:35.602195 19924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:49:35.602385 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "0d9479a6b0eb405d9d68f017b8da3422"
format_stamp: "Formatted at 2026-06-19 16:49:35 on dist-test-slave-qb36"
server_key: "53c7399a7cf9a615640c007348011f1e"
server_key_iv: "c2295155153f56fe17bd1b3ac1ebabf2"
server_key_version: "encryptionkey@0"
I20260619 16:49:35.602614 19924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:35.608225 19924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:35.609025 19924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:35.609297 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:35.610227 19924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:35.611725 19924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:35.611845 19924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:35.611974 19924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:35.612057 19924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:35.644390 19924 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:37639
I20260619 16:49:35.644476 20050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:37639 every 8 connection(s)
I20260619 16:49:35.645720 19924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:35.653682 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.648280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:44249 (local address 127.14.153.62:35443)
0619 16:49:35.648466 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.648471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.649872 (+ 1401us) server_negotiation.cc:407] Connection header received
0619 16:49:35.651570 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.651573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.651622 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.651710 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.652995 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.652999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.653002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.653019 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.653052 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.653064 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.653170 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.653421 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.653473 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:49:35.654608 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.648796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35443 (local address 127.14.153.3:44249)
0619 16:49:35.649688 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.649722 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.651336 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.651937 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.651944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.652033 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:35.652632 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:35.652634 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:35.652677 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:35.652756 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:35.653183 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:35.653191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:35.653197 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.653433 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.653631 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":271,"threads_started":1}
I20260619 16:49:35.655800 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 19924
I20260619 16:49:35.655970 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:35.656353 20051 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35443
I20260619 16:49:35.656528 14948 external_mini_cluster.cc:1428] Setting key 79ed13b056d38c3f4e262a59622b3534
I20260619 16:49:35.656687 20051 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:35.657382 20051 heartbeater.cc:511] Master 127.14.153.62:35443 requested a full tablet report, sending...
I20260619 16:49:35.658999 19603 ts_manager.cc:194] Registered new tserver with Master: 0d9479a6b0eb405d9d68f017b8da3422 (127.14.153.3:37639)
I20260619 16:49:35.672153 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:35.679162 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.676742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.14.153.62:35443)
0619 16:49:35.676976 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.676985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.677452 (+ 467us) server_negotiation.cc:407] Connection header received
0619 16:49:35.677583 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.677587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.677639 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.677717 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.678567 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.678571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.678574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.678588 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.678616 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.678627 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.678718 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.678750 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.678789 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:49:35.682880 19603 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:35.689772 19603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:35.709992 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.702625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33777 (local address 127.0.0.1:51024)
0619 16:49:35.703275 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.703310 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.703441 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.707562 (+ 4121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.707566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.707589 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:35.708395 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:35.708400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:35.708444 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:35.708539 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:35.708977 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:35.708987 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:35.708995 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.709178 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.709273 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":339,"threads_started":1}
I20260619 16:49:35.710084 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.703269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.14.153.1:33777)
0619 16:49:35.706289 (+ 3020us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.706300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.706336 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:49:35.706994 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.707003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.707396 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.707541 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.708666 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.708677 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.708692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.708720 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.708824 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.708849 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.708996 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.709313 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.709420 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2845,"thread_start_us":2721,"threads_started":1}
I20260619 16:49:35.710625 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.702098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:37639 (local address 127.0.0.1:43320)
0619 16:49:35.703779 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.703800 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.703889 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.707265 (+ 3376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.707275 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.707391 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:35.708313 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:35.708649 (+ 336us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:35.708668 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:35.708700 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:35.709237 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:35.709242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:35.709246 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.709416 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.709471 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":1373,"threads_started":1}
I20260619 16:49:35.711264 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.702391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.14.153.3:37639)
0619 16:49:35.706653 (+ 4262us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.706664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.706698 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:49:35.706766 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.706775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.707063 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.707943 (+ 880us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.709003 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.709010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.709022 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.709039 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.709096 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.709116 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.710056 (+ 940us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.710658 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.710896 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4102,"thread_start_us":3939,"threads_started":1}
I20260619 16:49:35.712414 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.704426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.14.153.2:39563)
0619 16:49:35.704951 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.704964 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.707130 (+ 2166us) server_negotiation.cc:407] Connection header received
0619 16:49:35.707396 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.707404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.707587 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.707800 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.711516 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.711524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.711543 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.711566 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.711608 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.711623 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.711731 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.711953 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.712052 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":239,"threads_started":1}
I20260619 16:49:35.712522 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.703894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:39563 (local address 127.0.0.1:52942)
0619 16:49:35.706993 (+ 3099us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.707010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.707214 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.709130 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.709138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.709164 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:35.711245 (+ 2081us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:35.711250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:35.711267 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:35.711312 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:35.711814 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:35.711817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:35.711820 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.712068 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.712114 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2981,"thread_start_us":2820,"threads_started":1}
I20260619 16:49:35.714329 19725 tablet_service.cc:1467] Processing CreateTablet for tablet 66b33db9657a4c09a4df92c90ee0f70b (DEFAULT_TABLE table=test-table [id=6f24bede2e6d4df6a465d115bc2584d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:35.714749 19986 tablet_service.cc:1467] Processing CreateTablet for tablet 66b33db9657a4c09a4df92c90ee0f70b (DEFAULT_TABLE table=test-table [id=6f24bede2e6d4df6a465d115bc2584d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:35.715904 19725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b33db9657a4c09a4df92c90ee0f70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:35.715502 19856 tablet_service.cc:1467] Processing CreateTablet for tablet 66b33db9657a4c09a4df92c90ee0f70b (DEFAULT_TABLE table=test-table [id=6f24bede2e6d4df6a465d115bc2584d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:35.716068 19986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b33db9657a4c09a4df92c90ee0f70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:35.716841 19856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b33db9657a4c09a4df92c90ee0f70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:35.728534 20069 tablet_bootstrap.cc:492] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: Bootstrap starting.
I20260619 16:49:35.730875 20070 tablet_bootstrap.cc:492] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: Bootstrap starting.
I20260619 16:49:35.732390 20071 tablet_bootstrap.cc:492] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987: Bootstrap starting.
I20260619 16:49:35.732906 20069 tablet_bootstrap.cc:654] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:35.734218 20069 log.cc:824] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:35.734577 20070 tablet_bootstrap.cc:654] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:35.735291 20071 tablet_bootstrap.cc:654] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:35.736099 20070 log.cc:824] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:35.736446 20071 log.cc:824] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:35.739344 20071 tablet_bootstrap.cc:492] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987: No bootstrap required, opened a new log
I20260619 16:49:35.739459 20069 tablet_bootstrap.cc:492] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: No bootstrap required, opened a new log
I20260619 16:49:35.739460 20070 tablet_bootstrap.cc:492] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: No bootstrap required, opened a new log
I20260619 16:49:35.739667 20071 ts_tablet_manager.cc:1397] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260619 16:49:35.739727 20069 ts_tablet_manager.cc:1397] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260619 16:49:35.739724 20070 ts_tablet_manager.cc:1397] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260619 16:49:35.746984 20071 raft_consensus.cc:348] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } }
I20260619 16:49:35.747454 20071 raft_consensus.cc:374] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:35.747550 20071 raft_consensus.cc:729] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345a48ba65dc47539b437adaa52ea987, State: Initialized, Role: FOLLOWER
I20260619 16:49:35.748328 20071 consensus_queue.cc:260] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } }
I20260619 16:49:35.749586 19790 heartbeater.cc:503] Master 127.14.153.62:35443 was elected leader, sending a full tablet report...
I20260619 16:49:35.750290 20071 ts_tablet_manager.cc:1428] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260619 16:49:35.750288 20070 raft_consensus.cc:348] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } }
I20260619 16:49:35.750288 20069 raft_consensus.cc:348] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } }
I20260619 16:49:35.750721 20069 raft_consensus.cc:374] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:35.750722 20070 raft_consensus.cc:374] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:35.750828 20070 raft_consensus.cc:729] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9479a6b0eb405d9d68f017b8da3422, State: Initialized, Role: FOLLOWER
I20260619 16:49:35.750823 20069 raft_consensus.cc:729] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169c231ada94464b810a09ea006c0c32, State: Initialized, Role: FOLLOWER
I20260619 16:49:35.751454 20069 consensus_queue.cc:260] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } }
I20260619 16:49:35.751452 20070 consensus_queue.cc:260] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } }
I20260619 16:49:35.752557 20051 heartbeater.cc:503] Master 127.14.153.62:35443 was elected leader, sending a full tablet report...
I20260619 16:49:35.753510 20070 ts_tablet_manager.cc:1428] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 16:49:35.755373 20069 ts_tablet_manager.cc:1428] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260619 16:49:35.756413 19921 heartbeater.cc:503] Master 127.14.153.62:35443 was elected leader, sending a full tablet report...
W20260619 16:49:35.759197 19791 tablet.cc:2369] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:35.765827 20076 raft_consensus.cc:482] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:35.766132 20076 raft_consensus.cc:504] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } }
I20260619 16:49:35.768052 20076 leader_election.cc:288] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 169c231ada94464b810a09ea006c0c32 (127.14.153.2:39563), 345a48ba65dc47539b437adaa52ea987 (127.14.153.1:33777)
I20260619 16:49:35.770907 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.768383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:39563 (local address 127.14.153.3:33723)
0619 16:49:35.768680 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.768700 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.768813 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.769373 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.769380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.769415 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:35.770091 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:35.770093 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:35.770111 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:35.770157 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:35.770521 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:35.770528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:35.770536 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.770646 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.770701 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260619 16:49:35.770988 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.768389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:33723 (local address 127.14.153.2:39563)
0619 16:49:35.768595 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.768603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.768793 (+ 190us) server_negotiation.cc:407] Connection header received
0619 16:49:35.768955 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.768963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.769039 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.769160 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.770316 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.770322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.770328 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.770351 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.770393 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.770410 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.770518 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.770642 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.770717 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:49:35.771641 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.768928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:55611 (local address 127.14.153.1:33777)
0619 16:49:35.769141 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.769147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.769798 (+ 651us) server_negotiation.cc:407] Connection header received
0619 16:49:35.769848 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.769851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.769893 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.769996 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.770959 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.770965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.770969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.770988 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.771024 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.771070 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.771188 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.771353 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.771417 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:49:35.771832 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.768711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33777 (local address 127.14.153.3:55611)
0619 16:49:35.769611 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.769625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.769711 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.770016 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.770021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.770046 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:35.770746 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:35.770749 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:35.770764 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:35.770805 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:35.771240 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:35.771244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:35.771249 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.771354 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.771400 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":555,"threads_started":1}
I20260619 16:49:35.772020 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b33db9657a4c09a4df92c90ee0f70b" candidate_uuid: "0d9479a6b0eb405d9d68f017b8da3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "169c231ada94464b810a09ea006c0c32" is_pre_election: true
I20260619 16:49:35.772606 19876 raft_consensus.cc:2454] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9479a6b0eb405d9d68f017b8da3422 in term 0.
I20260619 16:49:35.772696 19745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b33db9657a4c09a4df92c90ee0f70b" candidate_uuid: "0d9479a6b0eb405d9d68f017b8da3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345a48ba65dc47539b437adaa52ea987" is_pre_election: true
I20260619 16:49:35.773234 19745 raft_consensus.cc:2454] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9479a6b0eb405d9d68f017b8da3422 in term 0.
I20260619 16:49:35.774060 19940 leader_election.cc:302] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d9479a6b0eb405d9d68f017b8da3422, 169c231ada94464b810a09ea006c0c32, 345a48ba65dc47539b437adaa52ea987; no voters:
I20260619 16:49:35.774502 20076 raft_consensus.cc:2781] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:35.774652 20076 raft_consensus.cc:482] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:35.774772 20076 raft_consensus.cc:3037] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:35.777271 20076 raft_consensus.cc:504] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } }
I20260619 16:49:35.777927 20076 leader_election.cc:288] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [CANDIDATE]: Term 1 election: Requested vote from peers 169c231ada94464b810a09ea006c0c32 (127.14.153.2:39563), 345a48ba65dc47539b437adaa52ea987 (127.14.153.1:33777)
I20260619 16:49:35.778435 19745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b33db9657a4c09a4df92c90ee0f70b" candidate_uuid: "0d9479a6b0eb405d9d68f017b8da3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345a48ba65dc47539b437adaa52ea987"
I20260619 16:49:35.778443 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b33db9657a4c09a4df92c90ee0f70b" candidate_uuid: "0d9479a6b0eb405d9d68f017b8da3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "169c231ada94464b810a09ea006c0c32"
I20260619 16:49:35.778632 19745 raft_consensus.cc:3037] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:35.778644 19876 raft_consensus.cc:3037] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:35.782013 19745 raft_consensus.cc:2454] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9479a6b0eb405d9d68f017b8da3422 in term 1.
I20260619 16:49:35.782058 19876 raft_consensus.cc:2454] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9479a6b0eb405d9d68f017b8da3422 in term 1.
I20260619 16:49:35.782569 19940 leader_election.cc:302] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d9479a6b0eb405d9d68f017b8da3422, 169c231ada94464b810a09ea006c0c32, 345a48ba65dc47539b437adaa52ea987; no voters:
I20260619 16:49:35.782928 20076 raft_consensus.cc:2781] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:35.784327 20076 raft_consensus.cc:686] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 1 LEADER]: Becoming Leader. State: Replica: 0d9479a6b0eb405d9d68f017b8da3422, State: Running, Role: LEADER
I20260619 16:49:35.784909 20076 consensus_queue.cc:237] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } }
I20260619 16:49:35.790397 19604 catalog_manager.cc:5495] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d9479a6b0eb405d9d68f017b8da3422 (127.14.153.3). New cstate: current_term: 1 leader_uuid: "0d9479a6b0eb405d9d68f017b8da3422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9479a6b0eb405d9d68f017b8da3422" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:35.814944 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.812663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.14.153.3:37639)
0619 16:49:35.812844 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.812852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.812922 (+ 70us) server_negotiation.cc:407] Connection header received
0619 16:49:35.813195 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.813200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.813280 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.813410 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.814323 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.814328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.814331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.814346 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.814377 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.814389 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.814526 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.814633 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.814690 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:49:35.823913 19745 raft_consensus.cc:1261] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [term 1 FOLLOWER]: Refusing update from remote peer 0d9479a6b0eb405d9d68f017b8da3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:35.823911 19876 raft_consensus.cc:1261] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [term 1 FOLLOWER]: Refusing update from remote peer 0d9479a6b0eb405d9d68f017b8da3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:35.824882 20076 consensus_queue.cc:1035] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "169c231ada94464b810a09ea006c0c32" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:35.825186 20079 consensus_queue.cc:1035] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345a48ba65dc47539b437adaa52ea987" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:35.834947 20082 mvcc.cc:205] Tried to move back new op lower bound from 7298612329763753984 to 7298612329619988480. Current Snapshot: MvccSnapshot[applied={T|T < 7298612329763753984}]
I20260619 16:49:35.840217 20083 mvcc.cc:205] Tried to move back new op lower bound from 7298612329763753984 to 7298612329619988480. Current Snapshot: MvccSnapshot[applied={T|T < 7298612329763753984}]
I20260619 16:49:35.854558 19604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:35.854802 19604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:35.858539 19604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8ab416dff834c309b617b14379ab958: Sending DeleteTablet for 3 replicas of tablet 66b33db9657a4c09a4df92c90ee0f70b
I20260619 16:49:35.859989 19986 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b33db9657a4c09a4df92c90ee0f70b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:35 UTC) from {username='slave'} at 127.0.0.1:43320
I20260619 16:49:35.860006 19725 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b33db9657a4c09a4df92c90ee0f70b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:35 UTC) from {username='slave'} at 127.0.0.1:51024
I20260619 16:49:35.860100 19856 tablet_service.cc:1514] Processing DeleteTablet for tablet 66b33db9657a4c09a4df92c90ee0f70b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:35 UTC) from {username='slave'} at 127.0.0.1:52942
I20260619 16:49:35.861965 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 19663
I20260619 16:49:35.871369 20092 tablet_replica.cc:330] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987: stopping tablet replica
I20260619 16:49:35.872985 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.863393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.14.153.1:33777)
0619 16:49:35.863570 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.863579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.868469 (+ 4890us) server_negotiation.cc:407] Connection header received
0619 16:49:35.868702 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.868708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.868842 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.868959 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.872150 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.872155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.872160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.872181 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.872224 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.872240 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.872336 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.872582 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.872645 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:49:35.873641 20092 raft_consensus.cc:2229] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:35.874110 19785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:35.874814 20093 tablet_replica.cc:330] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: stopping tablet replica
I20260619 16:49:35.875509 20093 raft_consensus.cc:2229] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:35.875926 20094 tablet_replica.cc:330] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: stopping tablet replica
I20260619 16:49:35.876251 20093 raft_consensus.cc:2258] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:35.876317 20094 raft_consensus.cc:2229] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:35.876989 20094 raft_consensus.cc:2258] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:35.878875 20093 ts_tablet_manager.cc:1905] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:35.879344 20094 ts_tablet_manager.cc:1905] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:35.886247 20093 ts_tablet_manager.cc:1918] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:35.886453 20094 ts_tablet_manager.cc:1918] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:35.886592 20093 log.cc:1196] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/66b33db9657a4c09a4df92c90ee0f70b
I20260619 16:49:35.886631 20094 log.cc:1196] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/66b33db9657a4c09a4df92c90ee0f70b
I20260619 16:49:35.887241 20093 ts_tablet_manager.cc:1939] T 66b33db9657a4c09a4df92c90ee0f70b P 169c231ada94464b810a09ea006c0c32: Deleting consensus metadata
I20260619 16:49:35.887245 20094 ts_tablet_manager.cc:1939] T 66b33db9657a4c09a4df92c90ee0f70b P 0d9479a6b0eb405d9d68f017b8da3422: Deleting consensus metadata
I20260619 16:49:35.889097 19591 catalog_manager.cc:4841] TS 169c231ada94464b810a09ea006c0c32 (127.14.153.2:39563): tablet 66b33db9657a4c09a4df92c90ee0f70b (table test-table [id=6f24bede2e6d4df6a465d115bc2584d7]) successfully deleted
I20260619 16:49:35.889433 19591 catalog_manager.cc:4841] TS 0d9479a6b0eb405d9d68f017b8da3422 (127.14.153.3:37639): tablet 66b33db9657a4c09a4df92c90ee0f70b (table test-table [id=6f24bede2e6d4df6a465d115bc2584d7]) successfully deleted
I20260619 16:49:35.976326 20092 raft_consensus.cc:2258] T 66b33db9657a4c09a4df92c90ee0f70b P 345a48ba65dc47539b437adaa52ea987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:35.976600 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 19663
W20260619 16:49:35.994280 19589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.153.1:33777 (error 108)
I20260619 16:49:35.994518 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 19794
I20260619 16:49:35.996634 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.996091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33777 (local address 127.0.0.1:51052)
0619 16:49:35.996321 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.996460 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260619 16:49:35.996861 19589 catalog_manager.cc:4584] TS 345a48ba65dc47539b437adaa52ea987 (127.14.153.1:33777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66b33db9657a4c09a4df92c90ee0f70b: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33777: connect: Connection refused (error 111)
I20260619 16:49:35.997789 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.995444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.14.153.2:39563)
0619 16:49:35.995625 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.995633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.995658 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:49:35.995854 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.995860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.995973 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.996083 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:35.997175 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:35.997180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:35.997183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:35.997197 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:35.997226 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:35.997237 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:35.997347 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.997454 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.997496 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:49:35.998595 19916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:36.050704 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.050122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33777 (local address 127.0.0.1:51066)
0619 16:49:36.050387 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.050511 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 16:49:36.051836 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.051458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33777 (local address 127.0.0.1:51080)
0619 16:49:36.051615 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.051698 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:49:36.087406 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 19794
I20260619 16:49:36.109639 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 19924
I20260619 16:49:36.112320 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.111833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33777 (local address 127.0.0.1:51092)
0619 16:49:36.111993 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.112123 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:49:36.113296 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.110384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.14.153.3:37639)
0619 16:49:36.110584 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:36.110588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:36.110711 (+ 123us) server_negotiation.cc:407] Connection header received
0619 16:49:36.110964 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:36.110969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:36.111076 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:36.111186 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:36.112158 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:36.112164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:36.112168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:36.112184 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:36.112220 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:36.112234 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:36.112388 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:36.112942 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 16:49:36.113001 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:49:36.113422 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.113000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33777 (local address 127.0.0.1:51100)
0619 16:49:36.113180 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.113269 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 16:49:36.114017 20046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:36.193965 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.193457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33777 (local address 127.0.0.1:51104)
0619 16:49:36.193684 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.193786 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 16:49:36.195195 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.194637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33777 (local address 127.0.0.1:51108)
0619 16:49:36.194913 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.194983 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 16:49:36.234705 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 19924
I20260619 16:49:36.253784 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 19573
I20260619 16:49:36.254962 19634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:36.359361 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 19573
2026-06-19T16:49:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2600 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 16:49:36.400494 14948 mini_oidc.cc:152] Starting JWKS server
I20260619 16:49:36.403183 14948 webserver.cc:460] Webserver started at https://127.0.0.1:41479/ using document root <none> and password file <none>
I20260619 16:49:36.403501 14948 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:49:36.404135 14948 webserver.cc:460] Webserver started at http://127.0.0.1:36557/ using document root <none> and password file <none>
2026-06-19T16:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:36Z Disabled control of system clock
I20260619 16:49:36.430871 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34017
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:41707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34017
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41479/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/testchainca.cert with env {}
W20260619 16:49:36.753535 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:36.753829 20106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:36.753925 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:36.761205 20106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:36.761314 20106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:36.761351 20106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:36.761404 20106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:36.761447 20106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:49:36.761500 20106 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:49:36.761539 20106 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41479/jwks.json
I20260619 16:49:36.771834 20106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41479/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:36.773097 20106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:36.774353 20106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:36.786036 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:36.785953 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:36.786399 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:36.787258 20106 server_base.cc:1034] running on GCE node
I20260619 16:49:36.788093 20106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:36.789687 20106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:36.790936 20106 hybrid_clock.cc:648] HybridClock initialized: now 1781887776790891 us; error 39 us; skew 500 ppm
I20260619 16:49:36.791317 20106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:49:36.793578 20106 webserver.cc:460] Webserver started at http://127.14.153.62:35677/ using document root <none> and password file <none>
I20260619 16:49:36.794291 20106 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:36.794425 20106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:36.794735 20106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:36.797673 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "edff506008754e4b9edecd60489692f4"
format_stamp: "Formatted at 2026-06-19 16:49:36 on dist-test-slave-qb36"
server_key: "c3ad5ac34e0c6266f61db061af4976a5"
server_key_iv: "5d6e8044f3fffedfe5428baab92662ce"
server_key_version: "encryptionkey@0"
I20260619 16:49:36.798475 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "edff506008754e4b9edecd60489692f4"
format_stamp: "Formatted at 2026-06-19 16:49:36 on dist-test-slave-qb36"
server_key: "c3ad5ac34e0c6266f61db061af4976a5"
server_key_iv: "5d6e8044f3fffedfe5428baab92662ce"
server_key_version: "encryptionkey@0"
I20260619 16:49:36.803791 20106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:49:36.807550 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:36.809054 20106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:49:36.809254 20106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "edff506008754e4b9edecd60489692f4"
format_stamp: "Formatted at 2026-06-19 16:49:36 on dist-test-slave-qb36"
server_key: "c3ad5ac34e0c6266f61db061af4976a5"
server_key_iv: "5d6e8044f3fffedfe5428baab92662ce"
server_key_version: "encryptionkey@0"
I20260619 16:49:36.809480 20106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:36.833425 20106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:36.834309 20106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:36.834627 20106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:36.852807 20106 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:34017
I20260619 16:49:36.852856 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:34017 every 8 connection(s)
I20260619 16:49:36.854156 20106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:36.857872 20172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:36.860139 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 20106
I20260619 16:49:36.860373 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:36.861141 14948 external_mini_cluster.cc:1428] Setting key e98770e96426484cdc379a4b85635c8f
I20260619 16:49:36.869848 20172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4: Bootstrap starting.
I20260619 16:49:36.873137 20172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:36.874202 20172 log.cc:824] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:36.874459 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.863167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.14.153.62:34017)
0619 16:49:36.864122 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:36.864133 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:36.864176 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:49:36.865810 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:36.865832 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:36.866286 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:36.866697 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:36.867301 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.868625 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.869970 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.870655 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.870851 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:36.872136 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:36.872143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:36.872156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:36.872676 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:36.872766 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:36.872786 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:36.872911 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:36.873138 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 16:49:36.873475 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":319,"threads_started":1}
I20260619 16:49:36.877138 20172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4: No bootstrap required, opened a new log
I20260619 16:49:36.878119 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:36.884650 20172 raft_consensus.cc:348] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff506008754e4b9edecd60489692f4" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34017 } }
I20260619 16:49:36.885005 20172 raft_consensus.cc:374] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:36.885106 20172 raft_consensus.cc:729] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edff506008754e4b9edecd60489692f4, State: Initialized, Role: FOLLOWER
I20260619 16:49:36.885680 20172 consensus_queue.cc:260] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff506008754e4b9edecd60489692f4" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34017 } }
I20260619 16:49:36.885941 20172 raft_consensus.cc:388] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:36.886088 20172 raft_consensus.cc:482] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:36.886282 20172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:36.888458 20172 raft_consensus.cc:504] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff506008754e4b9edecd60489692f4" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34017 } }
I20260619 16:49:36.888942 20172 leader_election.cc:302] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edff506008754e4b9edecd60489692f4; no voters:
I20260619 16:49:36.889492 20172 leader_election.cc:288] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:36.889633 20177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:36.890442 20177 raft_consensus.cc:686] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [term 1 LEADER]: Becoming Leader. State: Replica: edff506008754e4b9edecd60489692f4, State: Running, Role: LEADER
I20260619 16:49:36.890957 20177 consensus_queue.cc:237] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff506008754e4b9edecd60489692f4" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34017 } }
I20260619 16:49:36.891696 20172 sys_catalog.cc:564] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:36.893927 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edff506008754e4b9edecd60489692f4. Latest consensus state: current_term: 1 leader_uuid: "edff506008754e4b9edecd60489692f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff506008754e4b9edecd60489692f4" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34017 } } }
I20260619 16:49:36.894294 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:36.894059 20178 sys_catalog.cc:455] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edff506008754e4b9edecd60489692f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff506008754e4b9edecd60489692f4" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34017 } } }
I20260619 16:49:36.894481 20178 sys_catalog.cc:458] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:36.896708 20184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:36.901750 20184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:36.910681 20184 catalog_manager.cc:1347] Generated new cluster ID: 4c206acdd9e9485e845d3dc8b55fa919
I20260619 16:49:36.910797 20184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:36.928747 20184 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:36.930052 20184 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:36.939134 20184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edff506008754e4b9edecd60489692f4: Generated new TSK 0
I20260619 16:49:36.940284 20184 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:49:36.987119 20202 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 16:49:36.987322 20202 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:49:36.988260 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.966841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.14.153.62:34017)
0619 16:49:36.967144 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:36.967151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:36.967590 (+ 439us) server_negotiation.cc:407] Connection header received
0619 16:49:36.967946 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:36.967950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:36.968037 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:36.968201 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:49:36.968824 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.969543 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.971717 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.972186 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.972307 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:36.972907 (+ 600us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:49:36.987736 (+ 14829us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:49:36.988038 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41950: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 16:49:36.988425 20175 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41950: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 16:49:36.990387 20201 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:36.966720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34017 (local address 127.0.0.1:41950)
0619 16:49:36.967461 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.967482 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:36.967670 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:36.968257 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:36.968263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:36.968277 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:49:36.968675 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:36.968687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.969678 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.969684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:36.971558 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:36.971574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.972378 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.972385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:36.972572 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:36.972648 (+ 76us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:49:36.988067 (+ 15419us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:49:36.988464 (+ 397us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.153.62:34017: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":375,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.153.62:34017: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 16:49:36.992487 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 20106
I20260619 16:49:36.993641 20167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:37.087100 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 20106
2026-06-19T16:49:37Z chronyd exiting
I20260619 16:49:37.137648 14948 test_util.cc:175] -----------------------------------------------
I20260619 16:49:37.137835 14948 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887734944185-14948-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (741 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 16:49:37.141618 14948 mini_oidc.cc:152] Starting JWKS server
I20260619 16:49:37.143425 14948 webserver.cc:460] Webserver started at https://127.0.0.1:37371/ using document root <none> and password file <none>
I20260619 16:49:37.143646 14948 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:49:37.144259 14948 webserver.cc:460] Webserver started at http://127.0.0.1:39381/ using document root <none> and password file <none>
2026-06-19T16:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:37Z Disabled control of system clock
I20260619 16:49:37.170624 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:38831
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:38831
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37371/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/testchainca.cert with env {}
W20260619 16:49:37.491053 20209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:37.491353 20209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:37.491439 20209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:37.498733 20209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:37.498847 20209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:37.498885 20209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:37.498941 20209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:37.498986 20209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:49:37.499099 20209 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:49:37.499148 20209 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37371/jwks.json
I20260619 16:49:37.509428 20209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:38831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:38831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37371/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:37.510700 20209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:37.512264 20209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:37.523996 20214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.524034 20217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.524080 20215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:37.524969 20209 server_base.cc:1034] running on GCE node
I20260619 16:49:37.525815 20209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:37.527395 20209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:37.528646 20209 hybrid_clock.cc:648] HybridClock initialized: now 1781887777528601 us; error 39 us; skew 500 ppm
I20260619 16:49:37.529038 20209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:49:37.534861 20209 webserver.cc:460] Webserver started at http://127.14.153.62:43479/ using document root <none> and password file <none>
I20260619 16:49:37.535655 20209 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:37.535797 20209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:37.536136 20209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:37.539134 20209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "070c5ebb11814f608948fcdd197041a1"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-qb36"
server_key: "2f485daa60c7ba8e7ffe15cbd4d5b857"
server_key_iv: "2b075ab1939719d5fb9182e93f68f0fc"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.539969 20209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "070c5ebb11814f608948fcdd197041a1"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-qb36"
server_key: "2f485daa60c7ba8e7ffe15cbd4d5b857"
server_key_iv: "2b075ab1939719d5fb9182e93f68f0fc"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.545440 20209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:37.548959 20223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.550391 20209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:37.550590 20209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "070c5ebb11814f608948fcdd197041a1"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-qb36"
server_key: "2f485daa60c7ba8e7ffe15cbd4d5b857"
server_key_iv: "2b075ab1939719d5fb9182e93f68f0fc"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.550807 20209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:37.564816 20209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:37.565675 20209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:37.565991 20209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:37.584628 20209 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:38831
I20260619 16:49:37.584646 20274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:38831 every 8 connection(s)
I20260619 16:49:37.586063 20209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:37.589126 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 20209
I20260619 16:49:37.589329 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:37.590085 14948 external_mini_cluster.cc:1428] Setting key 056277804aed90a455d43fe1feff927d
I20260619 16:49:37.590736 20275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:37.604830 20275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1: Bootstrap starting.
I20260619 16:49:37.605400 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.592134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.14.153.62:38831)
0619 16:49:37.593122 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.593133 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.593175 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:49:37.595342 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.595364 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.595872 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.596229 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.596852 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.598178 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.599993 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.601042 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.601237 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.602792 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.602800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.602824 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:37.603468 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.603580 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:37.603611 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.603749 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.603973 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.604332 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":378,"threads_started":1}
I20260619 16:49:37.608489 20275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:37.609393 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:37.609665 20275 log.cc:824] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:37.612609 20275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1: No bootstrap required, opened a new log
I20260619 16:49:37.620215 20275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070c5ebb11814f608948fcdd197041a1" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38831 } }
I20260619 16:49:37.620635 20275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:37.620740 20275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070c5ebb11814f608948fcdd197041a1, State: Initialized, Role: FOLLOWER
I20260619 16:49:37.621369 20275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070c5ebb11814f608948fcdd197041a1" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38831 } }
I20260619 16:49:37.621600 20275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:37.621742 20275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:37.621919 20275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:37.624142 20275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070c5ebb11814f608948fcdd197041a1" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38831 } }
I20260619 16:49:37.624647 20275 leader_election.cc:302] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070c5ebb11814f608948fcdd197041a1; no voters:
I20260619 16:49:37.625237 20275 leader_election.cc:288] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:37.625440 20280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:37.626847 20280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [term 1 LEADER]: Becoming Leader. State: Replica: 070c5ebb11814f608948fcdd197041a1, State: Running, Role: LEADER
I20260619 16:49:37.627431 20280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070c5ebb11814f608948fcdd197041a1" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38831 } }
I20260619 16:49:37.627707 20275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:37.630335 20281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070c5ebb11814f608948fcdd197041a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070c5ebb11814f608948fcdd197041a1" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38831 } } }
I20260619 16:49:37.630517 20282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070c5ebb11814f608948fcdd197041a1. Latest consensus state: current_term: 1 leader_uuid: "070c5ebb11814f608948fcdd197041a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070c5ebb11814f608948fcdd197041a1" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38831 } } }
I20260619 16:49:37.630780 20281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:37.630787 20282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:37.632656 20290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:37.637054 20290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:37.645121 20290 catalog_manager.cc:1347] Generated new cluster ID: c9b4e512370240c983283d7e98fe59da
I20260619 16:49:37.645246 20290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:37.665771 20290 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:37.667675 20290 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:37.684798 20290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070c5ebb11814f608948fcdd197041a1: Generated new TSK 0
I20260619 16:49:37.685901 20290 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:49:37.721159 20305 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 16:49:37.721333 20305 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:49:37.722573 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.702003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.14.153.62:38831)
0619 16:49:37.702212 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.702219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.702663 (+ 444us) server_negotiation.cc:407] Connection header received
0619 16:49:37.702888 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.702894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.702996 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.703217 (+ 221us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:49:37.703796 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.704971 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.706331 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.707065 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.707202 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.707707 (+ 505us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:49:37.721871 (+ 14164us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:49:37.722242 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 16:49:37.722682 20304 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:37.701864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:38831 (local address 127.0.0.1:57708)
0619 16:49:37.702555 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.702576 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.702706 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.703259 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.703263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.703274 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:49:37.703572 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.703580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.705116 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.705120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.706184 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.706196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.707244 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.707250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.707406 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.707443 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:49:37.722145 (+ 14702us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:49:37.722372 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.153.62:38831: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":402,"threads_started":1}
W20260619 16:49:37.722709 20278 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 16:49:37.724073 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 20209
I20260619 16:49:37.725231 20270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:37.822134 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 20209
2026-06-19T16:49:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (736 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 16:49:37.877782 14948 mini_oidc.cc:152] Starting JWKS server
I20260619 16:49:37.880199 14948 webserver.cc:460] Webserver started at https://127.0.0.1:38069/ using document root <none> and password file <none>
I20260619 16:49:37.880468 14948 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:49:37.881101 14948 webserver.cc:460] Webserver started at http://127.0.0.1:39333/ using document root <none> and password file <none>
2026-06-19T16:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:37Z Disabled control of system clock
I20260619 16:49:37.907328 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34443
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34443
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38069/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:49:38.227583 20312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:38.227959 20312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:38.228112 20312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:38.235498 20312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:38.235648 20312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:38.235733 20312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:38.235849 20312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:38.235929 20312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:49:38.236017 20312 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:49:38.236092 20312 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38069/jwks.json
I20260619 16:49:38.245563 20312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38069/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:38.246924 20312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:38.248731 20312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:38.260958 20317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:38.260994 20318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:38.260931 20320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:38.261831 20312 server_base.cc:1034] running on GCE node
I20260619 16:49:38.262590 20312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:38.264087 20312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:38.265295 20312 hybrid_clock.cc:648] HybridClock initialized: now 1781887778265240 us; error 44 us; skew 500 ppm
I20260619 16:49:38.265641 20312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:49:38.267930 20312 webserver.cc:460] Webserver started at http://127.14.153.62:42359/ using document root <none> and password file <none>
I20260619 16:49:38.268576 20312 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:38.268658 20312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:38.268922 20312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:38.271741 20312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "97c64952240349488b0199cc8598596c"
format_stamp: "Formatted at 2026-06-19 16:49:38 on dist-test-slave-qb36"
server_key: "c1c8265412f7fe7ead4e673a8dcd1a71"
server_key_iv: "b4d0b4ac38e09b72b94a5d4e3052dbf7"
server_key_version: "encryptionkey@0"
I20260619 16:49:38.272472 20312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "97c64952240349488b0199cc8598596c"
format_stamp: "Formatted at 2026-06-19 16:49:38 on dist-test-slave-qb36"
server_key: "c1c8265412f7fe7ead4e673a8dcd1a71"
server_key_iv: "b4d0b4ac38e09b72b94a5d4e3052dbf7"
server_key_version: "encryptionkey@0"
I20260619 16:49:38.277626 20312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:38.281128 20326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:38.282630 20312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 16:49:38.282830 20312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "97c64952240349488b0199cc8598596c"
format_stamp: "Formatted at 2026-06-19 16:49:38 on dist-test-slave-qb36"
server_key: "c1c8265412f7fe7ead4e673a8dcd1a71"
server_key_iv: "b4d0b4ac38e09b72b94a5d4e3052dbf7"
server_key_version: "encryptionkey@0"
I20260619 16:49:38.283073 20312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:38.321036 20312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:38.321941 20312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:38.322266 20312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:38.340764 20312 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:34443
I20260619 16:49:38.340777 20377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:34443 every 8 connection(s)
I20260619 16:49:38.342123 20312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:38.346029 20378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.346197 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 20312
I20260619 16:49:38.346412 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:38.347016 14948 external_mini_cluster.cc:1428] Setting key ebe20c7e38ddd45487644d10a7e7305b
I20260619 16:49:38.358182 20378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c: Bootstrap starting.
I20260619 16:49:38.360659 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.348884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.14.153.62:34443)
0619 16:49:38.349923 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.349942 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.349985 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:49:38.352109 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.352130 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.352631 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.353025 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:38.353584 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.354992 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.356364 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.357011 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.357207 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.358411 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:38.358422 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:38.358444 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:38.358966 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:38.359107 (+ 141us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:38.359137 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:38.359268 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.359449 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.359770 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":369,"threads_started":1}
I20260619 16:49:38.361639 20378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.362700 20378 log.cc:824] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:38.364493 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:38.365990 20378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c: No bootstrap required, opened a new log
I20260619 16:49:38.373977 20378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c64952240349488b0199cc8598596c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34443 } }
I20260619 16:49:38.374312 20378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.374394 20378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c64952240349488b0199cc8598596c, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.374884 20378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c64952240349488b0199cc8598596c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34443 } }
I20260619 16:49:38.375137 20378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:38.375241 20378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:38.375373 20378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.377418 20378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c64952240349488b0199cc8598596c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34443 } }
I20260619 16:49:38.377892 20378 leader_election.cc:302] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97c64952240349488b0199cc8598596c; no voters:
I20260619 16:49:38.378422 20378 leader_election.cc:288] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:38.378587 20383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:38.379330 20383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [term 1 LEADER]: Becoming Leader. State: Replica: 97c64952240349488b0199cc8598596c, State: Running, Role: LEADER
I20260619 16:49:38.379877 20383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c64952240349488b0199cc8598596c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34443 } }
I20260619 16:49:38.380582 20378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:38.383327 20385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97c64952240349488b0199cc8598596c. Latest consensus state: current_term: 1 leader_uuid: "97c64952240349488b0199cc8598596c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c64952240349488b0199cc8598596c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34443 } } }
I20260619 16:49:38.383786 20385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:38.387249 20384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97c64952240349488b0199cc8598596c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c64952240349488b0199cc8598596c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34443 } } }
I20260619 16:49:38.387598 20384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:38.388196 20392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:38.392616 20392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:38.400725 20392 catalog_manager.cc:1347] Generated new cluster ID: 500ed0ee91db4b138b60f6be867ab332
I20260619 16:49:38.400873 20392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:38.415122 20392 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:38.416433 20392 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:38.423344 20392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97c64952240349488b0199cc8598596c: Generated new TSK 0
I20260619 16:49:38.424257 20392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:38.447876 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.439141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.14.153.62:34443)
0619 16:49:38.439358 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.439365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.439838 (+ 473us) server_negotiation.cc:407] Connection header received
0619 16:49:38.440048 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.440056 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.440152 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.440344 (+ 192us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:49:38.440770 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.441887 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.443429 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.444085 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.444229 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.444837 (+ 608us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:49:38.447246 (+ 2409us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:49:38.447586 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 16:49:38.448009 20381 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 16:49:38.449638 20407 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:38.439097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34443 (local address 127.0.0.1:50812)
0619 16:49:38.439732 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.439747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.439874 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.440351 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.440354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.440365 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:49:38.440622 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.440630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.442074 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.442078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.443289 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.443301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.444286 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.444291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.444484 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.444564 (+ 80us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:49:38.447459 (+ 2895us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:49:38.449250 (+ 1791us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.153.62:34443: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":1499,"thread_start_us":354,"threads_started":1}
I20260619 16:49:38.450721 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 20312
I20260619 16:49:38.451910 20373 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 16:49:38.460193 20408 webserver.cc:533] Webserver: sslize failed:
E20260619 16:49:38.460290 20408 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:49:38.540201 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 20312
2026-06-19T16:49:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (717 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T16:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:38Z Disabled control of system clock
I20260619 16:49:38.622282 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:44939
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:40491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:44939
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/testchainca.cert with env {}
W20260619 16:49:38.949803 20413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:38.950117 20413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:38.950203 20413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:38.957520 20413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:38.957628 20413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:38.957667 20413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:38.957715 20413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:38.957782 20413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:49:38.957836 20413 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:49:38.957883 20413 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 16:49:38.967721 20413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:44939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:44939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:38.968990 20413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:38.970232 20413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:38.983568 20418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:38.983502 20421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:38.983542 20419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:38.984302 20413 server_base.cc:1034] running on GCE node
I20260619 16:49:38.985081 20413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:38.986423 20413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:38.987613 20413 hybrid_clock.cc:648] HybridClock initialized: now 1781887778987570 us; error 43 us; skew 500 ppm
I20260619 16:49:38.987974 20413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:49:38.990319 20413 webserver.cc:460] Webserver started at http://127.14.153.62:42659/ using document root <none> and password file <none>
I20260619 16:49:38.991079 20413 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:38.991175 20413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:38.991485 20413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:38.994319 20413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "ece92ec2d4994ad0be9a5be9f2df202c"
format_stamp: "Formatted at 2026-06-19 16:49:38 on dist-test-slave-qb36"
server_key: "56ff9f201ac0738104d8ee1f6e1c251d"
server_key_iv: "8d33aca5667bede5e8adea9c1169dd95"
server_key_version: "encryptionkey@0"
I20260619 16:49:38.995195 20413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "ece92ec2d4994ad0be9a5be9f2df202c"
format_stamp: "Formatted at 2026-06-19 16:49:38 on dist-test-slave-qb36"
server_key: "56ff9f201ac0738104d8ee1f6e1c251d"
server_key_iv: "8d33aca5667bede5e8adea9c1169dd95"
server_key_version: "encryptionkey@0"
I20260619 16:49:39.000305 20413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:49:39.003674 20427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:39.005013 20413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:39.005200 20413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "ece92ec2d4994ad0be9a5be9f2df202c"
format_stamp: "Formatted at 2026-06-19 16:49:38 on dist-test-slave-qb36"
server_key: "56ff9f201ac0738104d8ee1f6e1c251d"
server_key_iv: "8d33aca5667bede5e8adea9c1169dd95"
server_key_version: "encryptionkey@0"
I20260619 16:49:39.005430 20413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:39.022374 20413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:39.023330 20413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:39.023617 20413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:39.041782 20413 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:44939
I20260619 16:49:39.041801 20478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:44939 every 8 connection(s)
I20260619 16:49:39.043156 20413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:39.046901 20479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:39.051577 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 20413
I20260619 16:49:39.051779 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:39.052606 14948 external_mini_cluster.cc:1428] Setting key 7cd5b50a30ea59ab2ef2c43544360f37
I20260619 16:49:39.058425 20479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c: Bootstrap starting.
I20260619 16:49:39.062039 20479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:39.063251 20479 log.cc:824] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:39.066854 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:39.054599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.14.153.62:44939)
0619 16:49:39.055693 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:39.055705 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:39.055752 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:49:39.057434 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:39.057458 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:39.057962 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:39.058309 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:39.058918 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:39.060324 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:39.062201 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:39.062894 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:39.063114 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:39.064478 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:39.064489 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:39.064508 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:39.065115 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:39.065187 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:39.065212 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:39.065308 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:39.065590 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 16:49:39.065950 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":413,"threads_started":1}
I20260619 16:49:39.067168 20479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c: No bootstrap required, opened a new log
I20260619 16:49:39.070801 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:39.075970 20479 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece92ec2d4994ad0be9a5be9f2df202c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44939 } }
I20260619 16:49:39.076314 20479 raft_consensus.cc:374] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:39.076387 20479 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece92ec2d4994ad0be9a5be9f2df202c, State: Initialized, Role: FOLLOWER
I20260619 16:49:39.076879 20479 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece92ec2d4994ad0be9a5be9f2df202c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44939 } }
I20260619 16:49:39.077117 20479 raft_consensus.cc:388] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:39.077225 20479 raft_consensus.cc:482] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:39.077349 20479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:39.079903 20479 raft_consensus.cc:504] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece92ec2d4994ad0be9a5be9f2df202c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44939 } }
I20260619 16:49:39.080377 20479 leader_election.cc:302] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ece92ec2d4994ad0be9a5be9f2df202c; no voters:
I20260619 16:49:39.080924 20479 leader_election.cc:288] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:39.081075 20484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:39.081741 20484 raft_consensus.cc:686] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [term 1 LEADER]: Becoming Leader. State: Replica: ece92ec2d4994ad0be9a5be9f2df202c, State: Running, Role: LEADER
I20260619 16:49:39.082247 20484 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece92ec2d4994ad0be9a5be9f2df202c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44939 } }
I20260619 16:49:39.083098 20479 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:39.084920 20486 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece92ec2d4994ad0be9a5be9f2df202c. Latest consensus state: current_term: 1 leader_uuid: "ece92ec2d4994ad0be9a5be9f2df202c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece92ec2d4994ad0be9a5be9f2df202c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44939 } } }
I20260619 16:49:39.085251 20486 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:39.090605 20485 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ece92ec2d4994ad0be9a5be9f2df202c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece92ec2d4994ad0be9a5be9f2df202c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44939 } } }
I20260619 16:49:39.090919 20485 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c [sys.catalog]: This master's current role is: LEADER
W20260619 16:49:39.097918 20499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:39.098062 20499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:39.098186 20491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:39.100307 20491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:39.109598 20491 catalog_manager.cc:1347] Generated new cluster ID: 6169485576ec42129ec9d8c72b378849
I20260619 16:49:39.109728 20491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:39.121466 20491 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:39.122862 20491 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:39.131171 20491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ece92ec2d4994ad0be9a5be9f2df202c: Generated new TSK 0
I20260619 16:49:39.132021 20491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:39.569737 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:39.145459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.14.153.62:44939)
0619 16:49:39.145655 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:39.145662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:39.146177 (+ 515us) server_negotiation.cc:407] Connection header received
0619 16:49:39.146507 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:39.146514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:39.146606 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:39.146789 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:49:39.147222 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:39.148390 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:39.149750 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:39.150421 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:39.150549 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:39.151050 (+ 501us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:49:39.569187 (+418137us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:49:39.569514 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48228: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 16:49:39.569905 20482 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48228: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 16:49:39.570092 20508 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:39.145437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:44939 (local address 127.0.0.1:48228)
0619 16:49:39.146048 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:39.146087 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:39.146296 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:39.146793 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:39.146796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:39.146808 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:49:39.147078 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:39.147094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:39.148558 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:39.148562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:39.149614 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:39.149625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:39.150591 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:39.150596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:39.150720 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:39.150750 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:49:39.569475 (+418725us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:49:39.569723 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.153.62:44939: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":335,"threads_started":1}
I20260619 16:49:39.571449 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 20413
I20260619 16:49:39.572656 20474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:39.675606 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 20413
2026-06-19T16:49:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1117 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (423 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (407 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:40 dist-test-slave-qb36 krb5kdc[20523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:40 dist-test-slave-qb36 krb5kdc[20523](info): set up 2 sockets
Jun 19 16:49:40 dist-test-slave-qb36 krb5kdc[20523](info): commencing operation
krb5kdc: starting...
W20260619 16:49:41.596477 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:41 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:49:41.811178 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35135
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:42.130841 20536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:42.131247 20536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:42.131417 20536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:42.138636 20536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:42.138746 20536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:42.138849 20536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:42.138931 20536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:42.138969 20536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:42.149552 20536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:42.150934 20536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:42.152297 20536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:42.164677 20542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.164659 20544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.165295 20541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:42.165729 20536 server_base.cc:1034] running on GCE node
I20260619 16:49:42.166472 20536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:42.168013 20536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:42.169250 20536 hybrid_clock.cc:648] HybridClock initialized: now 1781887782169221 us; error 38 us; skew 500 ppm
I20260619 16:49:42.169571 20536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:42 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:42.172964 20536 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:49:42.174610 20536 webserver.cc:460] Webserver started at http://127.14.153.62:40241/ using document root <none> and password file <none>
I20260619 16:49:42.175459 20536 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:42.175611 20536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:42.175964 20536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:42.179257 20536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "45df632f667f42599c826283f0c6435e"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-qb36"
server_key: "bbb652668082aecfc68ea62e04c8eeb0"
server_key_iv: "aefc0a3991df002c327ef5cfd528b25c"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.180099 20536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "45df632f667f42599c826283f0c6435e"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-qb36"
server_key: "bbb652668082aecfc68ea62e04c8eeb0"
server_key_iv: "aefc0a3991df002c327ef5cfd528b25c"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.185623 20536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:42.189363 20551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.191012 20536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:42.191237 20536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "45df632f667f42599c826283f0c6435e"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-qb36"
server_key: "bbb652668082aecfc68ea62e04c8eeb0"
server_key_iv: "aefc0a3991df002c327ef5cfd528b25c"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.191457 20536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:42.226629 20536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:42.231360 20536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:42.231657 20536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:42.251078 20536 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35135
I20260619 16:49:42.251088 20602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35135 every 8 connection(s)
I20260619 16:49:42.252517 20536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:42.256497 20603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:42.261937 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 20536
I20260619 16:49:42.262171 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:42.262965 14948 external_mini_cluster.cc:1428] Setting key 919c784caaa884e5eca48c042ee2c49a
I20260619 16:49:42.268016 20603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e: Bootstrap starting.
I20260619 16:49:42.271819 20603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:42.273119 20603 log.cc:824] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:42.276652 20603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e: No bootstrap required, opened a new log
Jun 19 16:49:42 dist-test-slave-qb36 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:42.285074 20603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45df632f667f42599c826283f0c6435e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35135 } }
I20260619 16:49:42.285508 20603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:42.285652 20603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45df632f667f42599c826283f0c6435e, State: Initialized, Role: FOLLOWER
I20260619 16:49:42.286267 20603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45df632f667f42599c826283f0c6435e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35135 } }
I20260619 16:49:42.286532 20603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:42.286684 20603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:42.286885 20603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:42.287511 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.265159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.14.153.62:35135)
0619 16:49:42.266040 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:42.266053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:42.266102 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:49:42.267870 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:42.267902 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:42.268365 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:42.268702 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:42.270579 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.271993 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.273728 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.274782 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.274923 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.279612 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:42.279621 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:42.279637 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:42.279679 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:42.282864 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.283622 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.283629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.283636 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.283811 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.284214 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.284221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.284229 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.284770 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:42.285082 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:42.285565 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 16:49:42.286235 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":433,"threads_started":1}
I20260619 16:49:42.289350 20603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45df632f667f42599c826283f0c6435e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35135 } }
I20260619 16:49:42.289871 20603 leader_election.cc:302] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45df632f667f42599c826283f0c6435e; no voters:
I20260619 16:49:42.290557 20603 leader_election.cc:288] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:42.290671 20608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:42.291327 20608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [term 1 LEADER]: Becoming Leader. State: Replica: 45df632f667f42599c826283f0c6435e, State: Running, Role: LEADER
I20260619 16:49:42.291939 20608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45df632f667f42599c826283f0c6435e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35135 } }
I20260619 16:49:42.292815 20603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:42.294860 20610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45df632f667f42599c826283f0c6435e. Latest consensus state: current_term: 1 leader_uuid: "45df632f667f42599c826283f0c6435e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45df632f667f42599c826283f0c6435e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35135 } } }
I20260619 16:49:42.295395 20610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:42.295246 20609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45df632f667f42599c826283f0c6435e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45df632f667f42599c826283f0c6435e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35135 } } }
I20260619 16:49:42.295544 20609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:42.297179 20616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:42.302207 20616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:42.311443 20616 catalog_manager.cc:1347] Generated new cluster ID: 6659dab0f4134dd09230d07b12e2c2b6
I20260619 16:49:42.311563 20616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:42.348501 20616 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:42.349867 20616 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:42.355952 20616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45df632f667f42599c826283f0c6435e: Generated new TSK 0
I20260619 16:49:42.357026 20616 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:49:42.421206 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35135
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:42.761358 20629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:42.761700 20629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:42.761869 20629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:42.769328 20629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:42.769497 20629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:42.769667 20629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:42.779814 20629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:42.781188 20629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:42.782543 20629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:42.796792 20635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.796792 20637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.796958 20634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:42.797847 20629 server_base.cc:1034] running on GCE node
I20260619 16:49:42.798519 20629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:42.799674 20629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:42.800907 20629 hybrid_clock.cc:648] HybridClock initialized: now 1781887782800864 us; error 42 us; skew 500 ppm
I20260619 16:49:42.801240 20629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:42 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:42.807678 20629 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:49:42.809257 20629 webserver.cc:460] Webserver started at http://127.14.153.1:43347/ using document root <none> and password file <none>
I20260619 16:49:42.809991 20629 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:42.810099 20629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:42.810355 20629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:42.813408 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "8488ba79acd540d0bea3ee9e6942c484"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-qb36"
server_key: "6aeceaec1cfb21e1422dd41ff0c013af"
server_key_iv: "1edc389bb859d869191b890e49a1ceb6"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.814227 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "8488ba79acd540d0bea3ee9e6942c484"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-qb36"
server_key: "6aeceaec1cfb21e1422dd41ff0c013af"
server_key_iv: "1edc389bb859d869191b890e49a1ceb6"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.819720 20629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:49:42.823355 20644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.824844 20629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:42.824996 20629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "8488ba79acd540d0bea3ee9e6942c484"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-qb36"
server_key: "6aeceaec1cfb21e1422dd41ff0c013af"
server_key_iv: "1edc389bb859d869191b890e49a1ceb6"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.825178 20629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:42.858064 20629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:42.862597 20629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:42.862840 20629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:42.863806 20629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:42.865099 20629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:42.865172 20629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.865262 20629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:42.865304 20629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.899271 20629 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:42237
I20260619 16:49:42.899313 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:42237 every 8 connection(s)
I20260619 16:49:42.900597 20629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:42.904421 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 20629
I20260619 16:49:42.904625 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:42.905408 14948 external_mini_cluster.cc:1428] Setting key 40c6c0c636d10bcb6807fe35daea3985
Jun 19 16:49:42 dist-test-slave-qb36 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:42.923326 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.903235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:49667 (local address 127.14.153.62:35135)
0619 16:49:42.903699 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:42.903704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:42.904973 (+ 1269us) server_negotiation.cc:407] Connection header received
0619 16:49:42.907357 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:42.907361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:42.907474 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:42.907600 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:42.909177 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.910313 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.912039 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.912779 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.912933 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.917440 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:42.917446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:42.917449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:42.917482 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:42.920048 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.920935 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.920940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.920942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.921059 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.921505 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.921517 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.921519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.921783 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:42.921952 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:42.922687 (+ 735us) server_negotiation.cc:299] Negotiation successful
0619 16:49:42.922920 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260619 16:49:42.924467 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.903949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35135 (local address 127.14.153.1:49667)
0619 16:49:42.904820 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:42.904863 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:42.906946 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:42.907851 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:42.907860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:42.908397 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:42.908999 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.909016 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.910485 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.910488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.911894 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.911906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.913034 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.913040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.913434 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.914260 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:42.914300 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:42.917195 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:42.920230 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:42.920242 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:42.920263 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:42.920774 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:42.921180 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:42.921186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:42.921189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:42.921388 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:42.921914 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:42.921925 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:42.922303 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 16:49:42.922579 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 16:49:42.923272 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:42.926033 20757 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35135
I20260619 16:49:42.926384 20757 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:42.927232 20757 heartbeater.cc:511] Master 127.14.153.62:35135 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
I20260619 16:49:42.929777 20568 ts_manager.cc:194] Registered new tserver with Master: 8488ba79acd540d0bea3ee9e6942c484 (127.14.153.1:42237)
I20260619 16:49:42.931924 20568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:49667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:49:42.954089 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35135
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:43.280022 20763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:43.280305 20763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:43.280495 20763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:43.287915 20763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:43.288053 20763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:43.288242 20763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:43.298728 20763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:43.300037 20763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:43.301344 20763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:43.314360 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:43.314378 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:43.314589 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:43.315160 20763 server_base.cc:1034] running on GCE node
I20260619 16:49:43.315802 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:43.316732 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:43.317971 20763 hybrid_clock.cc:648] HybridClock initialized: now 1781887783317913 us; error 54 us; skew 500 ppm
I20260619 16:49:43.318284 20763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:43 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:43.322021 20763 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:49:43.323608 20763 webserver.cc:460] Webserver started at http://127.14.153.2:37027/ using document root <none> and password file <none>
I20260619 16:49:43.324270 20763 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:43.324368 20763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:43.324625 20763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:43.327605 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "941ec8923b48463f9b2af3e6709d997e"
format_stamp: "Formatted at 2026-06-19 16:49:43 on dist-test-slave-qb36"
server_key: "60b2b5ce1e912429c8fc7b1cf8cfbb8f"
server_key_iv: "24ae745fa189446252b2a08fca26bb39"
server_key_version: "encryptionkey@0"
I20260619 16:49:43.328356 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "941ec8923b48463f9b2af3e6709d997e"
format_stamp: "Formatted at 2026-06-19 16:49:43 on dist-test-slave-qb36"
server_key: "60b2b5ce1e912429c8fc7b1cf8cfbb8f"
server_key_iv: "24ae745fa189446252b2a08fca26bb39"
server_key_version: "encryptionkey@0"
I20260619 16:49:43.333734 20763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:49:43.337117 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.338467 20763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:43.338635 20763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "941ec8923b48463f9b2af3e6709d997e"
format_stamp: "Formatted at 2026-06-19 16:49:43 on dist-test-slave-qb36"
server_key: "60b2b5ce1e912429c8fc7b1cf8cfbb8f"
server_key_iv: "24ae745fa189446252b2a08fca26bb39"
server_key_version: "encryptionkey@0"
I20260619 16:49:43.338819 20763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:43.372581 20763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:43.377143 20763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:43.377449 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:43.378468 20763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:43.379930 20763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:43.380003 20763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.380168 20763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:43.380226 20763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.411762 20763 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:40751
I20260619 16:49:43.411803 20890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:40751 every 8 connection(s)
I20260619 16:49:43.413059 20763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:43.414696 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 20763
I20260619 16:49:43.414880 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:43.415342 14948 external_mini_cluster.cc:1428] Setting key 4a989fe434bb0e03e2d65136d2e591a5
Jun 19 16:49:43 dist-test-slave-qb36 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:43.437170 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.416062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:49219 (local address 127.14.153.62:35135)
0619 16:49:43.416241 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.416247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.417561 (+ 1314us) server_negotiation.cc:407] Connection header received
0619 16:49:43.420534 (+ 2973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.420539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.420610 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.420707 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.422904 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.423692 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.425868 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.426287 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.426426 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.429922 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:43.429933 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:43.429940 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:43.429989 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:43.433774 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:43.434489 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:43.434497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:43.434503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:43.434668 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:43.435219 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:43.435224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:43.435227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:43.435629 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:43.435783 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:43.436504 (+ 721us) server_negotiation.cc:299] Negotiation successful
0619 16:49:43.436745 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:43.437979 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.416582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35135 (local address 127.14.153.2:49219)
0619 16:49:43.417404 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.417470 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.420166 (+ 2696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.421001 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.421016 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.421833 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.422693 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.422722 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.423901 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.423910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.425718 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.425730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.426523 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.426531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.426766 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.427504 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.427530 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.429664 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:43.433936 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:43.433943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:43.433957 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:43.434362 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:43.434786 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:43.434789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:43.434791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:43.434988 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:43.435754 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:43.435759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:43.436101 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 16:49:43.436329 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 16:49:43.436709 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":306,"threads_started":1}
I20260619 16:49:43.439527 20891 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35135
I20260619 16:49:43.439857 20891 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
I20260619 16:49:43.440828 20891 heartbeater.cc:511] Master 127.14.153.62:35135 requested a full tablet report, sending...
I20260619 16:49:43.442402 20568 ts_manager.cc:194] Registered new tserver with Master: 941ec8923b48463f9b2af3e6709d997e (127.14.153.2:40751)
I20260619 16:49:43.443306 20568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:49219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:49:43.466768 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35135
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:43.797641 20896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:43.797951 20896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:43.798043 20896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:43.805315 20896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:43.805421 20896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:43.805532 20896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:43.815598 20896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:43.816860 20896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:43.818116 20896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:43.831595 20901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:43.831630 20904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:43.832273 20902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:43.832404 20896 server_base.cc:1034] running on GCE node
I20260619 16:49:43.833011 20896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:43.833956 20896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:43.835146 20896 hybrid_clock.cc:648] HybridClock initialized: now 1781887783835084 us; error 51 us; skew 500 ppm
I20260619 16:49:43.835465 20896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:43 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:43.838539 20896 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:49:43.840081 20896 webserver.cc:460] Webserver started at http://127.14.153.3:42991/ using document root <none> and password file <none>
I20260619 16:49:43.840792 20896 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:43.840940 20896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:43.841269 20896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:43.844424 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "d876aead9b3a4bc3934902b763e46dec"
format_stamp: "Formatted at 2026-06-19 16:49:43 on dist-test-slave-qb36"
server_key: "c160ef280179fe8ccf72a102220ebc39"
server_key_iv: "58667b4140f1f119eeb5bff11bcc3460"
server_key_version: "encryptionkey@0"
I20260619 16:49:43.845178 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "d876aead9b3a4bc3934902b763e46dec"
format_stamp: "Formatted at 2026-06-19 16:49:43 on dist-test-slave-qb36"
server_key: "c160ef280179fe8ccf72a102220ebc39"
server_key_iv: "58667b4140f1f119eeb5bff11bcc3460"
server_key_version: "encryptionkey@0"
I20260619 16:49:43.850343 20896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 16:49:43.853998 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.855409 20896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:49:43.855595 20896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "d876aead9b3a4bc3934902b763e46dec"
format_stamp: "Formatted at 2026-06-19 16:49:43 on dist-test-slave-qb36"
server_key: "c160ef280179fe8ccf72a102220ebc39"
server_key_iv: "58667b4140f1f119eeb5bff11bcc3460"
server_key_version: "encryptionkey@0"
I20260619 16:49:43.855830 20896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:43.872119 20896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:43.876164 20896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:43.876489 20896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:43.877430 20896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:43.878719 20896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:43.878832 20896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.878989 20896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:43.879078 20896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 16:49:43.912420 20896 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:43637
I20260619 16:49:43.912503 21023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:43637 every 8 connection(s)
I20260619 16:49:43.913666 20896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:43.916775 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 20896
I20260619 16:49:43.916942 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:43.917577 14948 external_mini_cluster.cc:1428] Setting key eb4ac5022b53d4a6e5588b2808249613
Jun 19 16:49:43 dist-test-slave-qb36 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:49:43.934983 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.916267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:34331 (local address 127.14.153.62:35135)
0619 16:49:43.916518 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.916527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.917922 (+ 1395us) server_negotiation.cc:407] Connection header received
0619 16:49:43.920187 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.920192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.920297 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.920386 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.922021 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.922782 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.924391 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.924854 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.924952 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.928621 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:43.928626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:43.928630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:43.928664 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:43.931770 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:43.932592 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:43.932597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:43.932601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:43.932798 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:43.933419 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:43.933423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:43.933425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:43.933689 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:43.933831 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:43.934548 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 16:49:43.934710 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:49:43.935214 20757 heartbeater.cc:503] Master 127.14.153.62:35135 was elected leader, sending a full tablet report...
I20260619 16:49:43.936165 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.916764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35135 (local address 127.14.153.3:34331)
0619 16:49:43.917727 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.917783 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.919861 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.920669 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.920678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.921295 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.921862 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.921883 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.922925 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.922929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.924257 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.924291 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.925039 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.925044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.925241 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.925941 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.925962 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.928279 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:43.931907 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:43.931916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:43.931935 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:43.932470 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:43.932997 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:43.933003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:43.933006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:43.933266 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:43.933807 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:43.933813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:43.934170 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 16:49:43.934406 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 16:49:43.934795 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":358,"threads_started":1}
I20260619 16:49:43.937564 21024 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35135
I20260619 16:49:43.937834 21024 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:43.938580 21024 heartbeater.cc:511] Master 127.14.153.62:35135 requested a full tablet report, sending...
I20260619 16:49:43.940052 20568 ts_manager.cc:194] Registered new tserver with Master: d876aead9b3a4bc3934902b763e46dec (127.14.153.3:43637)
I20260619 16:49:43.940896 20568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:34331
I20260619 16:49:43.945750 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/test-admin.keytab.
Jun 19 16:49:43 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:43.972548 14948 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 16:49:43.977592 21031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:43.975181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.14.153.1:42237)
0619 16:49:43.975748 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.975780 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.975807 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:49:43.975868 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.975876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.976090 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.976216 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.977214 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49026: BlockingRecv error: recv got EOF from 127.0.0.1:49026 (error 108)
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":281,"threads_started":1}
W20260619 16:49:43.977560 14948 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:49:43 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:43.979221 14948 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:49:43 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:43.980855 14948 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:49:43 dist-test-slave-qb36 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:43.996237 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.982537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.14.153.1:42237)
0619 16:49:43.982743 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.982749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.982867 (+ 118us) server_negotiation.cc:407] Connection header received
0619 16:49:43.982989 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.982993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.983135 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.983218 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.984164 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.985125 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.986306 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.986859 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.986965 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.990443 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:43.990455 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:43.990469 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:43.990501 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:43.993471 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:43.994204 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:43.994211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:43.994214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:43.994357 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:43.994713 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:43.994716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:43.994718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:43.995128 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:43.995416 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:43.995719 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 16:49:43.995963 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 16:49:44.002144 21031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:44.000639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.14.153.1:42237)
0619 16:49:44.000849 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.000855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.000998 (+ 143us) server_negotiation.cc:407] Connection header received
0619 16:49:44.001117 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.001120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.001177 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.001251 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.001900 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49044: BlockingRecv error: recv got EOF from 127.0.0.1:49044 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 16:49:44.002252 14948 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:44.003840 14948 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:44.005095 14948 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:44.019111 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:44.006323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.14.153.1:42237)
0619 16:49:44.006527 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.006534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.006791 (+ 257us) server_negotiation.cc:407] Connection header received
0619 16:49:44.006963 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.006968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.007075 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.007183 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.008121 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.008804 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.009994 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.010421 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.010563 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.014012 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:44.014018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:44.014023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:44.014063 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:44.016559 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.017188 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.017194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.017198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.017327 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.017700 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.017706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.017709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.017975 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:44.018226 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:44.018518 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 16:49:44.018707 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 16:49:44.023273 21031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:44.021519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.14.153.1:42237)
0619 16:49:44.021738 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.021746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.021990 (+ 244us) server_negotiation.cc:407] Connection header received
0619 16:49:44.022181 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.022185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.022267 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.022364 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.023021 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49062: BlockingRecv error: recv got EOF from 127.0.0.1:49062 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 16:49:44.023485 14948 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887734944185-14948-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:44.025008 14948 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:44.026564 14948 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:49:44.040792 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:44.028080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.14.153.1:42237)
0619 16:49:44.028298 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.028304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.028548 (+ 244us) server_negotiation.cc:407] Connection header received
0619 16:49:44.028718 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.028722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.028812 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.028893 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.030081 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.030790 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.032029 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.032434 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.032522 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.035965 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:44.035970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:44.035973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:44.036005 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:44.038414 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.039016 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.039019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.039022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.039225 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.039562 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.039565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.039567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.039810 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:44.040033 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:44.040270 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:49:44.040482 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 16:49:44.042434 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 20629
I20260619 16:49:44.054257 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:44.043167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.14.153.1:42237)
0619 16:49:44.043331 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.043339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.043649 (+ 310us) server_negotiation.cc:407] Connection header received
0619 16:49:44.043840 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.043845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.043915 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.044011 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.044901 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.045583 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.046777 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.047288 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.047455 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.049373 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:44.049379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:44.049384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:44.049425 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:44.051732 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.052371 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.052376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.052380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.052523 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.052898 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.052904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.052908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.053152 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:44.053424 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:44.053711 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 16:49:44.053933 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:49:44.054854 20752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:44.139068 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 20629
I20260619 16:49:44.155957 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 20763
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:49:44.171528 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:44.156916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.14.153.2:40751)
0619 16:49:44.157492 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.157499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.157525 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:49:44.157586 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.157592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.157821 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.157960 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.158862 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.160075 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.161292 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.161865 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.161965 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.165685 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:44.165697 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:44.165711 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:44.165782 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:44.168684 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.169488 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.169496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.169499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.169629 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.170038 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.170042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.170044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.170421 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:44.170656 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:44.170914 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 16:49:44.171215 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
I20260619 16:49:44.172851 20886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:44.257478 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 20763
I20260619 16:49:44.274686 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 20896
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:49:44.290642 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:44.275835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.14.153.3:43637)
0619 16:49:44.276420 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.276427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.276455 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:49:44.276517 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.276525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.276735 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.276930 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.278043 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.279105 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.280394 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.280955 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.281090 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.285035 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:44.285047 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:44.285062 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:44.285103 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:44.288087 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.288789 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.288796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.288798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.288922 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:44.289258 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:44.289262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:44.289264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:44.289593 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:44.289855 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:44.290060 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 16:49:44.290335 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":298,"threads_started":1}
I20260619 16:49:44.292086 21019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:44.374768 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 20896
I20260619 16:49:44.390885 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 20536
I20260619 16:49:44.392138 20598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:44.494204 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 20536
2026-06-19T16:49:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3990 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[21051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[21051](info): set up 2 sockets
Jun 19 16:49:44 dist-test-slave-qb36 krb5kdc[21051](info): commencing operation
krb5kdc: starting...
W20260619 16:49:45.601025 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:45 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.62.keytab.
Entry for principal oryx/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.62.keytab.
I20260619 16:49:45.810725 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:40135
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.62.keytab
--principal=oryx/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:40135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:46.133559 21064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:46.133970 21064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:46.134130 21064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:46.141534 21064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:46.141706 21064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:46.141830 21064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:46.141929 21064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:46.142015 21064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:46.151916 21064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:40135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.62.keytab
--principal=oryx/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:40135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:46.153355 21064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:46.154753 21064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:46.166762 21069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:46.166883 21070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:46.166880 21072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:46.167904 21064 server_base.cc:1034] running on GCE node
I20260619 16:49:46.168639 21064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:46.170183 21064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:46.171420 21064 hybrid_clock.cc:648] HybridClock initialized: now 1781887786171361 us; error 43 us; skew 500 ppm
I20260619 16:49:46.171753 21064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:46 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887786, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:46.174955 21064 init.cc:377] Logged in from keytab as oryx/127.14.153.62@KRBTEST.COM (short username oryx)
I20260619 16:49:46.176467 21064 webserver.cc:460] Webserver started at http://127.14.153.62:33721/ using document root <none> and password file <none>
I20260619 16:49:46.177129 21064 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:46.177228 21064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:46.177475 21064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:46.180477 21064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "d1416edc98814347a06ab1a14319c731"
format_stamp: "Formatted at 2026-06-19 16:49:46 on dist-test-slave-qb36"
server_key: "9788ff0e26f6892500dd6e43fb539ba5"
server_key_iv: "330431241c0452889ae334fb3f2f16f7"
server_key_version: "encryptionkey@0"
I20260619 16:49:46.181180 21064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "d1416edc98814347a06ab1a14319c731"
format_stamp: "Formatted at 2026-06-19 16:49:46 on dist-test-slave-qb36"
server_key: "9788ff0e26f6892500dd6e43fb539ba5"
server_key_iv: "330431241c0452889ae334fb3f2f16f7"
server_key_version: "encryptionkey@0"
I20260619 16:49:46.186240 21064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:49:46.189795 21079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:46.191345 21064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:49:46.191489 21064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "d1416edc98814347a06ab1a14319c731"
format_stamp: "Formatted at 2026-06-19 16:49:46 on dist-test-slave-qb36"
server_key: "9788ff0e26f6892500dd6e43fb539ba5"
server_key_iv: "330431241c0452889ae334fb3f2f16f7"
server_key_version: "encryptionkey@0"
I20260619 16:49:46.191648 21064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:46.207974 21064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:46.212389 21064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:46.212625 21064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:46.231480 21064 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:40135
I20260619 16:49:46.231503 21130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:40135 every 8 connection(s)
I20260619 16:49:46.232859 21064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:46.236618 21131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:46.241773 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 21064
I20260619 16:49:46.241972 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:46.242755 14948 external_mini_cluster.cc:1428] Setting key bda2d5240cdca30f2af74469d179b18f
I20260619 16:49:46.248139 21131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: Bootstrap starting.
I20260619 16:49:46.251222 21131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:46.252489 21131 log.cc:824] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:46.255975 21131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: No bootstrap required, opened a new log
Jun 19 16:49:46 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:46.263995 21131 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } }
I20260619 16:49:46.264380 21131 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:46.264504 21131 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1416edc98814347a06ab1a14319c731, State: Initialized, Role: FOLLOWER
I20260619 16:49:46.265234 21131 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } }
I20260619 16:49:46.265525 21131 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:46.265692 21131 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:46.265899 21131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:46.266552 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:46.244953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.14.153.62:40135)
0619 16:49:46.245783 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:46.245795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:46.245847 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:49:46.247558 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:46.247581 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:46.247985 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:46.248367 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:46.249876 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.251204 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.253060 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.254110 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.254255 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:46.258861 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:46.258870 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:46.258890 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:46.258939 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:46.262126 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:46.262846 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:46.262853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:46.262860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:46.263076 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:46.263477 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:46.263484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:46.263487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:46.264028 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:46.264335 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:46.264790 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 16:49:46.265199 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":366,"threads_started":1}
I20260619 16:49:46.268769 21131 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } }
I20260619 16:49:46.269317 21131 leader_election.cc:302] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1416edc98814347a06ab1a14319c731; no voters:
I20260619 16:49:46.270017 21131 leader_election.cc:288] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:46.270119 21136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:46.270797 21136 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 1 LEADER]: Becoming Leader. State: Replica: d1416edc98814347a06ab1a14319c731, State: Running, Role: LEADER
I20260619 16:49:46.271440 21136 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } }
I20260619 16:49:46.272251 21131 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:46.274225 21138 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1416edc98814347a06ab1a14319c731. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } }
I20260619 16:49:46.274580 21138 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:46.279326 21137 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } }
I20260619 16:49:46.279891 21137 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:46.280706 21145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:46.285780 21145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:46.295261 21145 catalog_manager.cc:1347] Generated new cluster ID: 7ed1fb63941147198e32fd16bf7627d8
I20260619 16:49:46.295421 21145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:46.310511 21145 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:46.312470 21145 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:46.330607 21145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: Generated new TSK 0
I20260619 16:49:46.331696 21145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab.
Entry for principal oryx/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab.
I20260619 16:49:46.390813 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40135
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab
--principal=oryx/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:46.719836 21157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:46.720113 21157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:46.720274 21157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:46.727620 21157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:46.727737 21157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:46.727957 21157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:46.737825 21157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab
--principal=oryx/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:46.739203 21157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:46.740504 21157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:46.753472 21162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:46.753553 21165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:46.753517 21163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:46.755097 21157 server_base.cc:1034] running on GCE node
I20260619 16:49:46.755744 21157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:46.756683 21157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:46.757922 21157 hybrid_clock.cc:648] HybridClock initialized: now 1781887786757879 us; error 60 us; skew 500 ppm
I20260619 16:49:46.758283 21157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:46 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887786, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:46.761523 21157 init.cc:377] Logged in from keytab as oryx/127.14.153.1@KRBTEST.COM (short username oryx)
I20260619 16:49:46.763161 21157 webserver.cc:460] Webserver started at http://127.14.153.1:41347/ using document root <none> and password file <none>
I20260619 16:49:46.763883 21157 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:46.764031 21157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:46.764346 21157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:46.767405 21157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "1494c019676b4562a447e90ea1f42977"
format_stamp: "Formatted at 2026-06-19 16:49:46 on dist-test-slave-qb36"
server_key: "a0eb9bc22b86d8c18c8ccc7ab80e3bca"
server_key_iv: "18f755663eb5e28fc9474fac5baba565"
server_key_version: "encryptionkey@0"
I20260619 16:49:46.768178 21157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "1494c019676b4562a447e90ea1f42977"
format_stamp: "Formatted at 2026-06-19 16:49:46 on dist-test-slave-qb36"
server_key: "a0eb9bc22b86d8c18c8ccc7ab80e3bca"
server_key_iv: "18f755663eb5e28fc9474fac5baba565"
server_key_version: "encryptionkey@0"
I20260619 16:49:46.773650 21157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:46.777410 21172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:46.778861 21157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:46.779121 21157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "1494c019676b4562a447e90ea1f42977"
format_stamp: "Formatted at 2026-06-19 16:49:46 on dist-test-slave-qb36"
server_key: "a0eb9bc22b86d8c18c8ccc7ab80e3bca"
server_key_iv: "18f755663eb5e28fc9474fac5baba565"
server_key_version: "encryptionkey@0"
I20260619 16:49:46.779330 21157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:46.804442 21157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:46.808511 21157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:46.808761 21157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:46.809630 21157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:46.811013 21157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:46.811151 21157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:46.811239 21157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:46.811285 21157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:46.844200 21157 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:42227
I20260619 16:49:46.844244 21284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:42227 every 8 connection(s)
I20260619 16:49:46.845485 21157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:46.851402 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 21157
I20260619 16:49:46.851613 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:46.852273 14948 external_mini_cluster.cc:1428] Setting key 8ac1b1e801acf2eba6a6e650922411e0
Jun 19 16:49:46 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887786, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:46.866370 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:46.848095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:44179 (local address 127.14.153.62:40135)
0619 16:49:46.848511 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:46.848516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:46.849473 (+ 957us) server_negotiation.cc:407] Connection header received
0619 16:49:46.851247 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:46.851254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:46.851345 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:46.851472 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:46.853000 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.853971 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.855590 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.855986 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.856123 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:46.859850 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:46.859855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:46.859859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:46.859890 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:46.862999 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:46.863784 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:46.863792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:46.863798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:46.863940 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:46.864402 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:46.864409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:46.864415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:46.864774 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:46.864931 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:46.865644 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 16:49:46.865892 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":214,"threads_started":1}
I20260619 16:49:46.867105 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:46.848635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.1:44179)
0619 16:49:46.849306 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:46.849357 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:46.850924 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:46.851756 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:46.851764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:46.852235 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:46.852836 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:46.852853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.854130 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.854139 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:46.855444 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:46.855455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.856223 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.856230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:46.856471 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:46.857166 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:46.857189 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:46.859621 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:46.863175 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:46.863183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:46.863197 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:46.863636 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:46.864077 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:46.864081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:46.864084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:46.864278 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:46.864892 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:46.864898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:46.865244 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 16:49:46.865477 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 16:49:46.865896 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":260,"threads_started":1}
I20260619 16:49:46.868644 21285 heartbeater.cc:347] Connected to a master server at 127.14.153.62:40135
I20260619 16:49:46.868970 21285 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:46.869879 21285 heartbeater.cc:511] Master 127.14.153.62:40135 requested a full tablet report, sending...
I20260619 16:49:46.872205 21096 ts_manager.cc:194] Registered new tserver with Master: 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227)
WARNING: no policy specified for oryx/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:46.873759 21096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:44179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab.
Entry for principal oryx/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab.
I20260619 16:49:46.902176 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40135
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab
--principal=oryx/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:47.223528 21291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:47.223892 21291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:47.224066 21291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:47.231750 21291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:47.231902 21291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:47.232077 21291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:47.242034 21291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab
--principal=oryx/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:47.243333 21291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:47.244645 21291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:47.257931 21296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:47.257889 21297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:47.258230 21299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:47.258754 21291 server_base.cc:1034] running on GCE node
I20260619 16:49:47.259482 21291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:47.260445 21291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:47.261708 21291 hybrid_clock.cc:648] HybridClock initialized: now 1781887787261648 us; error 44 us; skew 500 ppm
I20260619 16:49:47.262051 21291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:47 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887787, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:47.265255 21291 init.cc:377] Logged in from keytab as oryx/127.14.153.2@KRBTEST.COM (short username oryx)
I20260619 16:49:47.266712 21291 webserver.cc:460] Webserver started at http://127.14.153.2:41147/ using document root <none> and password file <none>
I20260619 16:49:47.267606 21291 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:47.267831 21291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:47.268097 21291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:47.270922 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "f65afa21f9194948ab94de36d244d1dd"
format_stamp: "Formatted at 2026-06-19 16:49:47 on dist-test-slave-qb36"
server_key: "78e317658a696516524be53cf429d7ad"
server_key_iv: "bb2c12011960e99ff05e869cb25be902"
server_key_version: "encryptionkey@0"
I20260619 16:49:47.271711 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "f65afa21f9194948ab94de36d244d1dd"
format_stamp: "Formatted at 2026-06-19 16:49:47 on dist-test-slave-qb36"
server_key: "78e317658a696516524be53cf429d7ad"
server_key_iv: "bb2c12011960e99ff05e869cb25be902"
server_key_version: "encryptionkey@0"
I20260619 16:49:47.276788 21291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:49:47.280081 21306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:47.281466 21291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:47.281625 21291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "f65afa21f9194948ab94de36d244d1dd"
format_stamp: "Formatted at 2026-06-19 16:49:47 on dist-test-slave-qb36"
server_key: "78e317658a696516524be53cf429d7ad"
server_key_iv: "bb2c12011960e99ff05e869cb25be902"
server_key_version: "encryptionkey@0"
I20260619 16:49:47.281813 21291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:47.306499 21291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:47.310555 21291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:47.310853 21291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:47.311898 21291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:47.313215 21291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:47.313323 21291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:47.313463 21291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:47.313532 21291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:47.347365 21291 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:44093
I20260619 16:49:47.347396 21418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:44093 every 8 connection(s)
I20260619 16:49:47.348630 21291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:47.352524 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 21291
I20260619 16:49:47.352696 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:47.353184 14948 external_mini_cluster.cc:1428] Setting key 52c93d4fa0434f3c7861cf16de03fd87
Jun 19 16:49:47 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887787, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:47.368376 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:47.351226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:33151 (local address 127.14.153.62:40135)
0619 16:49:47.351401 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:47.351405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:47.352588 (+ 1183us) server_negotiation.cc:407] Connection header received
0619 16:49:47.354498 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:47.354503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:47.354587 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:47.354710 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:47.356252 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.356955 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.358539 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.358965 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.359138 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.362540 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:47.362545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:47.362548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:47.362579 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:47.365336 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:47.366058 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:47.366071 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:47.366073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:47.366192 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:47.366616 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:47.366623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:47.366626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:47.366898 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:47.367067 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:47.367777 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 16:49:47.368030 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:49:47.369362 21421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:47.351726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.2:33151)
0619 16:49:47.352429 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:47.352467 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:47.354220 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:47.354944 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:47.354952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:47.355510 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:47.356085 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.356103 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.357118 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.357123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.358376 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.358388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.359228 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.359236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.359474 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.360185 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:47.360208 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:47.362303 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:47.365475 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:47.365484 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:47.365503 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:47.365930 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:47.366300 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:47.366305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:47.366316 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:47.366505 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:47.367055 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:47.367064 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:47.367407 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 16:49:47.367686 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 16:49:47.368226 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":288,"threads_started":1}
I20260619 16:49:47.371101 21419 heartbeater.cc:347] Connected to a master server at 127.14.153.62:40135
I20260619 16:49:47.371408 21419 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:47.372128 21419 heartbeater.cc:511] Master 127.14.153.62:40135 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:47.373847 21096 ts_manager.cc:194] Registered new tserver with Master: f65afa21f9194948ab94de36d244d1dd (127.14.153.2:44093)
I20260619 16:49:47.374691 21096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:33151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab.
Entry for principal oryx/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab.
I20260619 16:49:47.402841 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40135
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab
--principal=oryx/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:47.735210 21424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:47.735502 21424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:47.735651 21424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:47.743052 21424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:47.743160 21424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:47.743386 21424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:47.753978 21424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab
--principal=oryx/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:40135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:47.755358 21424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:47.756634 21424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:47.769840 21430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:47.769891 21429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:47.769917 21432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:47.770820 21424 server_base.cc:1034] running on GCE node
I20260619 16:49:47.771519 21424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:47.772588 21424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:47.773792 21424 hybrid_clock.cc:648] HybridClock initialized: now 1781887787773753 us; error 43 us; skew 500 ppm
I20260619 16:49:47.774115 21424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:47 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887787, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:47.777803 21424 init.cc:377] Logged in from keytab as oryx/127.14.153.3@KRBTEST.COM (short username oryx)
I20260619 16:49:47.779392 21424 webserver.cc:460] Webserver started at http://127.14.153.3:44145/ using document root <none> and password file <none>
I20260619 16:49:47.780181 21424 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:47.780342 21424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:47.780637 21424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:47.784368 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "d6c6e5f441b74d4a9c9b889a030dc609"
format_stamp: "Formatted at 2026-06-19 16:49:47 on dist-test-slave-qb36"
server_key: "e2bf2c97ef66c4c7a4f6d35a41a48867"
server_key_iv: "bb6c2d08077d7a0eec6548a92102e7bb"
server_key_version: "encryptionkey@0"
I20260619 16:49:47.785493 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "d6c6e5f441b74d4a9c9b889a030dc609"
format_stamp: "Formatted at 2026-06-19 16:49:47 on dist-test-slave-qb36"
server_key: "e2bf2c97ef66c4c7a4f6d35a41a48867"
server_key_iv: "bb6c2d08077d7a0eec6548a92102e7bb"
server_key_version: "encryptionkey@0"
I20260619 16:49:47.792037 21424 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260619 16:49:47.795588 21439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:47.796985 21424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:49:47.797147 21424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "d6c6e5f441b74d4a9c9b889a030dc609"
format_stamp: "Formatted at 2026-06-19 16:49:47 on dist-test-slave-qb36"
server_key: "e2bf2c97ef66c4c7a4f6d35a41a48867"
server_key_iv: "bb6c2d08077d7a0eec6548a92102e7bb"
server_key_version: "encryptionkey@0"
I20260619 16:49:47.797330 21424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:47.819778 21424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:47.824106 21424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:47.824398 21424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:47.825356 21424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:47.826644 21424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:47.826716 21424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:47.826877 21424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:47.826926 21424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:47.862635 21424 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:46809
I20260619 16:49:47.862674 21551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:46809 every 8 connection(s)
I20260619 16:49:47.863940 21424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:47.873685 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 21424
I20260619 16:49:47.873874 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:47.874323 14948 external_mini_cluster.cc:1428] Setting key c89506bdc54ceeed8edcf9706b8ea24d
I20260619 16:49:47.876968 21285 heartbeater.cc:503] Master 127.14.153.62:40135 was elected leader, sending a full tablet report...
Jun 19 16:49:47 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887787, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:47.884637 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:47.866467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:51407 (local address 127.14.153.62:40135)
0619 16:49:47.866653 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:47.866657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:47.867859 (+ 1202us) server_negotiation.cc:407] Connection header received
0619 16:49:47.869730 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:47.869735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:47.869823 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:47.869947 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:47.871463 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.872249 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.874554 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.875203 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.875348 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.879085 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:47.879091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:47.879096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:47.879157 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:47.881651 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:47.882354 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:47.882358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:47.882360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:47.882479 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:47.882965 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:47.882968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:47.882970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:47.883370 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:47.883484 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:47.884247 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 16:49:47.884401 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:49:47.885659 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:47.866970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.3:51407)
0619 16:49:47.867695 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:47.867742 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:47.869459 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:47.870188 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:47.870196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:47.870678 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:47.871301 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.871320 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.872403 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.872409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.874331 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.874343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.875430 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.875439 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.875740 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.876448 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:47.876475 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:47.878859 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:47.881806 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:47.881812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:47.881826 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:47.882249 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:47.882627 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:47.882632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:47.882634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:47.882855 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:47.883478 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:47.883484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:47.883921 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 16:49:47.884165 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 16:49:47.884564 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":269,"threads_started":1}
I20260619 16:49:47.887246 21552 heartbeater.cc:347] Connected to a master server at 127.14.153.62:40135
I20260619 16:49:47.887576 21552 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:47.888520 21552 heartbeater.cc:511] Master 127.14.153.62:40135 requested a full tablet report, sending...
I20260619 16:49:47.890218 21096 ts_manager.cc:194] Registered new tserver with Master: d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809)
I20260619 16:49:47.891080 21096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:51407
I20260619 16:49:47.902704 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.153.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.61.keytab.
Entry for principal oryx/127.14.153.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.61.keytab.
I20260619 16:49:47.948553 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.61:37409
--webserver_interface=127.14.153.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.61.keytab
--principal=oryx/127.14.153.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:40135,127.14.153.61:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:48.272028 21557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:48.272334 21557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:48.272477 21557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:48.279974 21557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:48.280092 21557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:48.280174 21557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:48.280261 21557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:48.280330 21557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:48.290707 21557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:40135,127.14.153.61:37409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.61.keytab
--principal=oryx/127.14.153.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.61:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.153.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
W20260619 16:49:48.291189 21557 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.153.62:40135,127.14.153.61:37409'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 16:49:48.292379 21557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:48.293653 21557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:48.305558 21562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:48.305625 21565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:48.305589 21563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:48.306802 21557 server_base.cc:1034] running on GCE node
I20260619 16:49:48.307616 21557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:48.308988 21557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:48.310210 21557 hybrid_clock.cc:648] HybridClock initialized: now 1781887788310148 us; error 35 us; skew 500 ppm
I20260619 16:49:48.310552 21557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:48 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887788, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:48.313709 21557 init.cc:377] Logged in from keytab as oryx/127.14.153.61@KRBTEST.COM (short username oryx)
I20260619 16:49:48.315194 21557 webserver.cc:460] Webserver started at http://127.14.153.61:38021/ using document root <none> and password file <none>
I20260619 16:49:48.315860 21557 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:48.315945 21557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:48.316197 21557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:48.319092 21557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data/instance:
uuid: "eb37ccfbcfd841eb9a5e502f2bd87749"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "e2d1074547df6668e6d9794c8bdcbe1c"
server_key_iv: "38306e7d324b8da7885d206aff6b73cf"
server_key_version: "encryptionkey@0"
I20260619 16:49:48.319868 21557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal/instance:
uuid: "eb37ccfbcfd841eb9a5e502f2bd87749"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "e2d1074547df6668e6d9794c8bdcbe1c"
server_key_iv: "38306e7d324b8da7885d206aff6b73cf"
server_key_version: "encryptionkey@0"
I20260619 16:49:48.325024 21557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:48.328477 21572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:48.329995 21557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:48.330168 21557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal
uuid: "eb37ccfbcfd841eb9a5e502f2bd87749"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "e2d1074547df6668e6d9794c8bdcbe1c"
server_key_iv: "38306e7d324b8da7885d206aff6b73cf"
server_key_version: "encryptionkey@0"
I20260619 16:49:48.330386 21557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:48.355898 21557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:48.360288 21557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:48.360558 21557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:48.377488 21419 heartbeater.cc:503] Master 127.14.153.62:40135 was elected leader, sending a full tablet report...
I20260619 16:49:48.379781 21557 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.61:37409
I20260619 16:49:48.379789 21623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.61:37409 every 8 connection(s)
I20260619 16:49:48.381155 21557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
I20260619 16:49:48.385368 21624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:48.388753 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 21557
I20260619 16:49:48.393172 21624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:49:48 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.61@KRBTEST.COM
Jun 19 16:49:48 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887788, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.61@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:48.411490 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.396849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.14.153.62:40135)
0619 16:49:48.397394 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.397401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.397870 (+ 469us) server_negotiation.cc:407] Connection header received
0619 16:49:48.398112 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.398116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.398180 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.398296 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.399476 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.400203 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.401699 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.402177 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.402305 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.405773 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:48.405778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:48.405781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:48.405812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:48.408612 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.409291 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.409298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.409301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.409426 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.409819 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.409825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.409829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.410110 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:48.410268 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:48.410875 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 16:49:48.411111 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":311,"threads_started":1}
I20260619 16:49:48.412021 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.391019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.14.153.61:37409)
0619 16:49:48.391865 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.391880 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.391931 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:49:48.393970 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.394002 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.394597 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.395104 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.396817 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.398624 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.399927 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.400656 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.400919 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.405365 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:48.405374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:48.405386 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:48.405421 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:48.407996 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.408639 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.408646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.408652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.408805 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.409190 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.409195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.409204 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.409680 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:48.410000 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:48.410439 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 16:49:48.410852 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":318,"threads_started":1}
I20260619 16:49:48.412076 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.396791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:55798)
0619 16:49:48.397723 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:48.397765 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:48.397955 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:48.398351 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:48.398359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:48.399022 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:48.399328 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.399336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.400343 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.400348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.401555 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.401566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.402409 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.402416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.402563 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.403262 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:48.403286 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:48.405564 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:48.408750 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.408763 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.408770 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.409171 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.409543 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.409548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.409552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.409704 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.410246 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:48.410254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:48.410421 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 16:49:48.410613 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:49:48.411063 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":49,"thread_start_us":556,"threads_started":1}
I20260619 16:49:48.414108 21624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:49:48 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887788, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.61@KRBTEST.COM for oryx/127.14.153.61@KRBTEST.COM
I20260619 16:49:48.426342 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.414959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.0.0.1:50864)
0619 16:49:48.415150 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:48.415166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:48.415257 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:48.415525 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:48.415529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:48.415855 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:48.415987 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.415995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.416917 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.416920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.417815 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.417826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.418508 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.418513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.418673 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.419305 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:48.419321 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:48.421275 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:48.424117 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.424121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.424124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.424493 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.424798 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.424802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.424804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.424923 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.425427 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:48.425430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:48.425547 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 16:49:48.425726 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:49:48.425965 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:49:48.426343 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.414818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.14.153.61:37409)
0619 16:49:48.415021 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.415053 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.415238 (+ 185us) server_negotiation.cc:407] Connection header received
0619 16:49:48.415362 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.415365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.415416 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.415491 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.416112 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.416790 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.417969 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.418372 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.418477 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.421438 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:48.421442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:48.421445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:48.421472 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:48.423985 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.424593 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.424597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.424599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.424705 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.425011 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.425013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.425015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.425280 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:48.425389 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:48.425794 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 16:49:48.426085 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":78}
I20260619 16:49:48.431303 21624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Bootstrap starting.
I20260619 16:49:48.434461 21624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:48.435645 21624 log.cc:824] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:48.438457 21624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: No bootstrap required, opened a new log
I20260619 16:49:48.445636 21624 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } }
I20260619 16:49:48.446022 21624 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:48.446131 21624 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb37ccfbcfd841eb9a5e502f2bd87749, State: Initialized, Role: FOLLOWER
I20260619 16:49:48.446784 21624 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } }
I20260619 16:49:48.447674 21631 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } } }
I20260619 16:49:48.448007 21631 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:48.448706 21624 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:49:48.457427 21642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:48.457584 21642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:48.460019 21557 master_runner.cc:428] Detected that this master eb37ccfbcfd841eb9a5e502f2bd87749 is joining an existing cluster
I20260619 16:49:48.460119 21557 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.153.61:37409
I20260619 16:49:48.486065 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.464586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.14.153.61:37409)
0619 16:49:48.464776 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.464782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.465504 (+ 722us) server_negotiation.cc:407] Connection header received
0619 16:49:48.465675 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.465680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.465768 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.465881 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.467824 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.469362 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.472452 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.473437 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.473567 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.477654 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:48.477662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:48.477667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:48.477707 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:48.482644 (+ 4937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.483612 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.483617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.483620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.483757 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.484233 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.484238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.484248 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.484675 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:48.484833 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:48.485371 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 16:49:48.485600 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:49:48.487059 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.464585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.0.0.1:50870)
0619 16:49:48.465386 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:48.465406 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:48.465551 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:48.466091 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:48.466099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:48.467073 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:48.467603 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.467618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.470035 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.470044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.472259 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.472281 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.473993 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.474002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.474257 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.475073 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:48.475109 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:48.477385 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:48.482794 (+ 5409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.482798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.482802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.483497 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.483892 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.483896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.483900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.484055 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.484907 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:48.484911 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:48.485084 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 16:49:48.486273 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0619 16:49:48.486567 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":416,"threads_started":1}
I20260619 16:49:48.492123 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.464553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.14.153.62:40135)
0619 16:49:48.464739 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.464747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.478069 (+ 13322us) server_negotiation.cc:407] Connection header received
0619 16:49:48.478345 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.478353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.478479 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.478537 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.479995 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.480711 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.482813 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.483501 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.483602 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.486056 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:48.486064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:48.486068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:48.486109 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:48.489433 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.490101 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.490107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.490110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.490253 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.490629 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.490637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.490642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.490931 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:48.491116 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:48.491517 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 16:49:48.491746 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:49:48.492424 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.465057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:55814)
0619 16:49:48.477960 (+ 12903us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:48.477984 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:48.478192 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:48.478793 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:48.478799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:48.479455 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:48.479839 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.479852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.480874 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.480881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.482644 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.482654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.483911 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.483917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.484105 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.484714 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:48.484739 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:48.485840 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:48.489615 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.489619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.489621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.489972 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.490372 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.490377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.490381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.490519 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.491082 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:48.491086 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:48.491261 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 16:49:48.491403 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:49:48.491729 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12057,"thread_start_us":11335,"threads_started":1}
WARNING: no policy specified for oryx/127.14.153.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.60@KRBTEST.COM" created.
Jun 19 16:49:48 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887786, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.62@KRBTEST.COM for oryx/127.14.153.61@KRBTEST.COM
I20260619 16:49:48.513525 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.498235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.14.153.61:37409)
0619 16:49:48.498435 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.498443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.500482 (+ 2039us) server_negotiation.cc:407] Connection header received
0619 16:49:48.500809 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.500813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.500901 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.501008 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.502240 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.502948 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.504264 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.504744 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.504890 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.508055 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:48.508060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:48.508063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:48.508092 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:48.510755 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.511540 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.511546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.511550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.511687 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.512080 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.512090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.512092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.512333 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:48.512493 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:48.512933 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 16:49:48.513149 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:49:48.513588 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.498235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.0.0.1:50886)
0619 16:49:48.500338 (+ 2103us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:48.500388 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:48.500609 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:48.501132 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:48.501141 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:48.501857 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:48.502072 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.502081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.503124 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.503129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.504120 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.504131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.504909 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.504915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.505099 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.505732 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:48.505748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:48.507866 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:48.510936 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.510948 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.510956 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.511418 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.511804 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.511811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.511817 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.511966 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.512480 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:48.512489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:48.512654 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 16:49:48.512837 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 16:49:48.513150 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1888,"thread_start_us":988,"threads_started":1}
I20260619 16:49:48.515792 21096 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.153.61:37409: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749"
member_type: NON_VOTER
last_known_addr {
host: "127.14.153.61"
port: 37409
}
attrs {
promote: true
}
}
dest_uuid: "d1416edc98814347a06ab1a14319c731"
cas_config_opid_index: -1
I20260619 16:49:48.516727 21096 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } }
I20260619 16:49:48.519905 21655 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER eb37ccfbcfd841eb9a5e502f2bd87749 (127.14.153.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } }
I20260619 16:49:48.521350 21599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:48.523396 21655 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.153.61:37409
I20260619 16:49:48.523844 21655 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } }
I20260619 16:49:48.524041 21655 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:48.524289 21557 master.cc:537] Master@127.14.153.61:37409 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.60.keytab.
Entry for principal oryx/127.14.153.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.60.keytab.
I20260619 16:49:48.526731 21599 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Refusing update from remote peer d1416edc98814347a06ab1a14319c731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 16:49:48.528160 21655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:48.528286 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.60:46829
--webserver_interface=127.14.153.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.60.keytab
--principal=oryx/127.14.153.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:40135,127.14.153.61:37409,127.14.153.60:46829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:48.530766 21081 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 16:49:48.532536 21081 consensus_peers.cc:487] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 -> Peer eb37ccfbcfd841eb9a5e502f2bd87749 (127.14.153.61:37409): Couldn't send request to peer eb37ccfbcfd841eb9a5e502f2bd87749. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:49:48.545332 21557 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:48.545778 21557 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:48.545980 21557 tablet_replica.cc:330] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: stopping tablet replica
I20260619 16:49:48.563019 21557 master.cc:552] Master@127.14.153.61:37409 shutdown complete.
I20260619 16:49:48.563194 21557 master_runner.cc:305] Clearing existing system tablet
I20260619 16:49:48.563553 21557 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:48.563669 21557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:48.571624 21557 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.007s
I20260619 16:49:48.573379 21660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:48.573776 21557 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:48.574079 21557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:48.574162 21557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal
uuid: "eb37ccfbcfd841eb9a5e502f2bd87749"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "e2d1074547df6668e6d9794c8bdcbe1c"
server_key_iv: "38306e7d324b8da7885d206aff6b73cf"
server_key_version: "encryptionkey@0"
I20260619 16:49:48.577932 21557 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:48.582731 21557 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:49:48.582921 21557 log.cc:1196] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 16:49:48.583460 21557 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Deleting consensus metadata
I20260619 16:49:48.583763 21557 master_runner.cc:315] Copying system tablet from 127.14.153.62:40135
I20260619 16:49:48.592382 21557 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: tablet copy: Beginning tablet copy session from remote peer at address 127.14.153.62:40135
I20260619 16:49:48.605482 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.593192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.14.153.62:40135)
0619 16:49:48.593409 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.593419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.593996 (+ 577us) server_negotiation.cc:407] Connection header received
0619 16:49:48.594200 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.594205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.594300 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.594423 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.595366 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.596257 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.597577 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.598082 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.598223 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.600033 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:48.600039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:48.600044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:48.600083 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:48.602837 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.603547 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.603554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.603557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.603679 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:48.604063 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:48.604068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:48.604071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:48.604353 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:48.604500 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:48.604879 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 16:49:48.605083 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:49:48.605551 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.593186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:55824)
0619 16:49:48.593873 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:48.593894 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:48.594061 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:48.594447 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:48.594452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:48.594891 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:48.595218 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.595228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.596383 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.596391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.597421 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.597432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.598237 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.598243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.598438 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.599137 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:48.599154 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:48.599839 (+ 685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:48.602971 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.602978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.602983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.603429 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.603790 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:48.603802 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:48.603806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:48.603950 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:48.604466 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:48.604470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:48.604629 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 16:49:48.604764 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 16:49:48.605098 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":378,"threads_started":1}
I20260619 16:49:48.606595 21116 tablet_copy_service.cc:140] P d1416edc98814347a06ab1a14319c731: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer eb37ccfbcfd841eb9a5e502f2bd87749 ({username='oryx', principal='oryx/127.14.153.61@KRBTEST.COM'} at 127.0.0.1:55824)
I20260619 16:49:48.606853 21116 tablet_copy_service.cc:161] P d1416edc98814347a06ab1a14319c731: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer eb37ccfbcfd841eb9a5e502f2bd87749 at {username='oryx', principal='oryx/127.14.153.61@KRBTEST.COM'} at 127.0.0.1:55824: session id = eb37ccfbcfd841eb9a5e502f2bd87749-00000000000000000000000000000000
I20260619 16:49:48.609908 21116 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:49:48.611828 21557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:49:48.616497 21557 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: tablet copy: Starting download of 0 data blocks...
I20260619 16:49:48.616804 21557 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: tablet copy: Starting download of 1 WAL segments...
I20260619 16:49:48.619145 21557 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:49:48.622241 21116 tablet_copy_service.cc:342] P d1416edc98814347a06ab1a14319c731: Request end of tablet copy session eb37ccfbcfd841eb9a5e502f2bd87749-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.153.61@KRBTEST.COM'} at 127.0.0.1:55824
I20260619 16:49:48.622433 21116 tablet_copy_service.cc:433] P d1416edc98814347a06ab1a14319c731: ending tablet copy session eb37ccfbcfd841eb9a5e502f2bd87749-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer eb37ccfbcfd841eb9a5e502f2bd87749
I20260619 16:49:48.632375 21557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:48.635193 21667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:48.647814 21668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:48.649822 21557 server_base.cc:1034] running on GCE node
W20260619 16:49:48.650028 21670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:48.650427 21557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:48.651141 21557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:48.652366 21557 hybrid_clock.cc:648] HybridClock initialized: now 1781887788652308 us; error 35 us; skew 500 ppm
I20260619 16:49:48.652595 21557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:48 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887788, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:48.654508 21557 init.cc:377] Logged in from keytab as oryx/127.14.153.61@KRBTEST.COM (short username oryx)
I20260619 16:49:48.656214 21557 webserver.cc:460] Webserver started at http://127.14.153.61:46379/ using document root <none> and password file <none>
I20260619 16:49:48.656565 21557 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:48.656663 21557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:48.659133 21557 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:49:48.660923 21677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:48.661396 21557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:48.661546 21557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal
uuid: "eb37ccfbcfd841eb9a5e502f2bd87749"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "e2d1074547df6668e6d9794c8bdcbe1c"
server_key_iv: "38306e7d324b8da7885d206aff6b73cf"
server_key_version: "encryptionkey@0"
I20260619 16:49:48.661710 21557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:48.672430 21557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:48.676031 21557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:48.706774 21557 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.61:37409
I20260619 16:49:48.706805 21728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.61:37409 every 8 connection(s)
I20260619 16:49:48.707696 21557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
I20260619 16:49:48.714673 21729 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:49:48.717209 21729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Bootstrap starting.
I20260619 16:49:48.729555 21729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:49:48.730007 21729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Bootstrap complete.
I20260619 16:49:48.730971 21729 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } }
I20260619 16:49:48.731249 21729 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb37ccfbcfd841eb9a5e502f2bd87749, State: Initialized, Role: LEARNER
I20260619 16:49:48.731478 21729 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } }
I20260619 16:49:48.732254 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } }
I20260619 16:49:48.732571 21729 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:48.732678 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:49:48.740272 21743 catalog_manager.cc:1259] Loaded cluster ID: 7ed1fb63941147198e32fd16bf7627d8
I20260619 16:49:48.740412 21743 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: loading cluster ID for follower catalog manager: success
I20260619 16:49:48.742765 21743 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: acquiring CA information for follower catalog manager: success
I20260619 16:49:48.744976 21743 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:49:48.893534 21552 heartbeater.cc:503] Master 127.14.153.62:40135 was elected leader, sending a full tablet report...
W20260619 16:49:48.949211 21658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:48.949512 21658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:48.949645 21658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:48.957662 21658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:48.957782 21658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:48.957870 21658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:48.957975 21658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:48.958035 21658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:48.968446 21658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:40135,127.14.153.61:37409,127.14.153.60:46829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.60.keytab
--principal=oryx/127.14.153.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.60:46829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.153.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:48.969791 21658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:48.971652 21658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:48.983949 21749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:48.983978 21748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:48.984016 21751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:48.984664 21658 server_base.cc:1034] running on GCE node
I20260619 16:49:48.985491 21658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:48.986917 21658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:48.988518 21658 hybrid_clock.cc:648] HybridClock initialized: now 1781887788988080 us; error 37 us; skew 500 ppm
I20260619 16:49:48.988890 21658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:48 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887788, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:48.992039 21658 init.cc:377] Logged in from keytab as oryx/127.14.153.60@KRBTEST.COM (short username oryx)
I20260619 16:49:48.993535 21658 webserver.cc:460] Webserver started at http://127.14.153.60:40603/ using document root <none> and password file <none>
I20260619 16:49:48.994217 21658 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:48.994369 21658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:48.994732 21658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:48.997865 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data/instance:
uuid: "a3a0303ea1b140869540f1fb602489b0"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "6a358974d9dddc5377c965040d295f76"
server_key_iv: "48323cbdf15f7fac9c595e9bd675f318"
server_key_version: "encryptionkey@0"
I20260619 16:49:48.998627 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal/instance:
uuid: "a3a0303ea1b140869540f1fb602489b0"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "6a358974d9dddc5377c965040d295f76"
server_key_iv: "48323cbdf15f7fac9c595e9bd675f318"
server_key_version: "encryptionkey@0"
I20260619 16:49:49.004140 21658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 16:49:49.007455 21758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:49.008807 21658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:49:49.008991 21658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal
uuid: "a3a0303ea1b140869540f1fb602489b0"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "6a358974d9dddc5377c965040d295f76"
server_key_iv: "48323cbdf15f7fac9c595e9bd675f318"
server_key_version: "encryptionkey@0"
I20260619 16:49:49.009187 21658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:49.019933 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.013829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.0.0.1:50894)
0619 16:49:49.014018 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.014040 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.014171 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.014949 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.014956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.014985 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:49.015217 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.015226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.016651 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.016657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.017903 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.017915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.019091 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.019097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.019363 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.019404 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.019516 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.019654 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 16:49:49.019970 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.013993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.14.153.61:37409)
0619 16:49:49.014604 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.014609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.014625 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:49:49.014682 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.014693 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.014780 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.014915 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:49.015390 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.016502 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.018059 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.018879 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.019025 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.019291 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.019541 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.019666 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":370,"threads_started":1}
I20260619 16:49:49.021111 21704 raft_consensus.cc:1203] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 16:49:49.021879 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1416edc98814347a06ab1a14319c731. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } }
I20260619 16:49:49.022145 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:49:49.037392 21658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:49.041882 21658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:49.042114 21658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:49.062023 21658 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.60:46829
I20260619 16:49:49.062080 21810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.60:46829 every 8 connection(s)
I20260619 16:49:49.063644 21658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
I20260619 16:49:49.065444 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 21658
I20260619 16:49:49.068102 21811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:49.076686 21811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:49:49 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.60@KRBTEST.COM
Jun 19 16:49:49 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887788, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.60@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:49.090684 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.068011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.14.153.60:46829)
0619 16:49:49.068920 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.068938 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.068984 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:49:49.070533 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.070554 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.070956 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.071425 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.073192 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.074682 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.076377 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.077094 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.077225 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.083014 (+ 5789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.083021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.083087 (+ 66us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.083129 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.086487 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.087226 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.087237 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.087246 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.087463 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.087812 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.087818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.087822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.088453 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.088657 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.089278 (+ 621us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.089704 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":470,"threads_started":1}
I20260619 16:49:49.095300 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.079759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.14.153.62:40135)
0619 16:49:49.079991 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.080000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.080605 (+ 605us) server_negotiation.cc:407] Connection header received
0619 16:49:49.081206 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.081216 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.081303 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.081404 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.082718 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.083554 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.085265 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.085884 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.086023 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.089720 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.089728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.089732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.089764 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.092546 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.093343 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.093351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.093354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.093486 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.093858 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.093863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.093866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.094128 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.094286 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.094708 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.094921 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:49:49.095386 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.079642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:55832)
0619 16:49:49.080450 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.080505 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.081065 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.081464 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.081469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.082240 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.082553 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.082572 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.083679 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.083685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.085057 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.085077 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.086062 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.086066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.086271 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.086898 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.086915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.089525 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:49.092670 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.092679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.092684 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.093220 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.093604 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.093608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.093612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.093729 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.094256 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:49.094263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:49.094434 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.094608 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.094921 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":387,"threads_started":1}
I20260619 16:49:49.096525 21811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:49:49 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887788, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.60@KRBTEST.COM for oryx/127.14.153.61@KRBTEST.COM
I20260619 16:49:49.108565 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.097165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.0.0.1:50898)
0619 16:49:49.097330 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.097344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.097417 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.097815 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.097818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.098491 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.098610 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.098617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.099657 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.099660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.100534 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.100543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.101355 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.101363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.101552 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.102102 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.102117 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.104030 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:49.106657 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.106660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.106663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.107005 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.107378 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.107381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.107383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.107495 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.107951 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:49.107954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:49.108062 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.108162 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.108350 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:49:49.108628 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.097379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50898 (local address 127.14.153.61:37409)
0619 16:49:49.097554 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.097558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.097573 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:49:49.097620 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.097623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.097701 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.097780 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.098773 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.099540 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.100652 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.101192 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.101291 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.104202 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.104207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.104210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.104240 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.106548 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.107151 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.107154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.107157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.107284 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.107581 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.107584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.107586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.107857 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.107965 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.108247 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.108406 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:49:49.109680 21811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:49:49 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887788, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.60@KRBTEST.COM for oryx/127.14.153.60@KRBTEST.COM
I20260619 16:49:49.122488 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.110460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:46829 (local address 127.0.0.1:58204)
0619 16:49:49.110663 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.110682 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.110844 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.111195 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.111199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.111545 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.111673 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.111680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.112605 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.112613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.113450 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.113460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.114231 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.114237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.114558 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.115142 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.115161 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.117064 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:49.119757 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.119764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.119769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.120100 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.120466 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.120473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.120479 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.120617 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.121089 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:49.121094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:49.121250 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.121382 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.121640 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 16:49:49.122572 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.110498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.14.153.60:46829)
0619 16:49:49.110663 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.110669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.110784 (+ 115us) server_negotiation.cc:407] Connection header received
0619 16:49:49.110947 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.110952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.111011 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.111182 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.111811 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.112485 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.113591 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.114075 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.114227 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.117228 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.117236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.117244 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.117269 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.119624 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.120233 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.120237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.120243 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.120365 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.120727 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.120735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.120738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.120980 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.121127 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.121450 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.122260 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":49}
I20260619 16:49:49.127826 21811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Bootstrap starting.
I20260619 16:49:49.130947 21811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:49.132091 21811 log.cc:824] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:49.135007 21811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: No bootstrap required, opened a new log
I20260619 16:49:49.142942 21811 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } }
I20260619 16:49:49.143368 21811 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:49.143503 21811 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a0303ea1b140869540f1fb602489b0, State: Initialized, Role: FOLLOWER
I20260619 16:49:49.144984 21811 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } }
I20260619 16:49:49.146212 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } } }
I20260619 16:49:49.146725 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:49.148183 21811 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:49:49.160104 21828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:49.160297 21828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:49.165079 21658 master_runner.cc:428] Detected that this master a3a0303ea1b140869540f1fb602489b0 is joining an existing cluster
I20260619 16:49:49.165244 21658 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.153.60:46829
I20260619 16:49:49.197917 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 21157
I20260619 16:49:49.228041 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.175601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.14.153.60:46829)
0619 16:49:49.175839 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.175848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.203542 (+ 27694us) server_negotiation.cc:407] Connection header received
0619 16:49:49.205726 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.205731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.205786 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.205853 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.210560 (+ 4707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.211843 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.215187 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.216038 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.216210 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.219242 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.219270 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.219279 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.219342 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.223699 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.225746 (+ 2047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.225751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.225755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.225906 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.226310 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.226315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.226318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.226756 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.226895 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.227396 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.227607 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":71}
I20260619 16:49:49.229358 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.175683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:46829 (local address 127.0.0.1:58216)
0619 16:49:49.203444 (+ 27761us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.203465 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.205612 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.206026 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.206031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.206716 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.210443 (+ 3727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.210453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.212081 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.212089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.214985 (+ 2896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.215006 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.216287 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.216310 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.216564 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.217370 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.217394 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.218668 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:49.223833 (+ 5165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.223838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.223843 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.224347 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.226034 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.226039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.226042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.226184 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.226958 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:49.226963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:49.227138 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.228597 (+ 1459us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.228889 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27597,"thread_start_us":38137,"threads_started":1}
Jun 19 16:49:49 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.1@KRBTEST.COM
I20260619 16:49:49.243193 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.177006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.14.153.61:37409)
0619 16:49:49.177345 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.177350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.216771 (+ 39421us) server_negotiation.cc:407] Connection header received
0619 16:49:49.219203 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.219212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.219310 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.219390 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.221209 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.222518 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.224972 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.225848 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.225979 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.228997 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.229007 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.229014 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.229062 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.235883 (+ 6821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.239474 (+ 3591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.239483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.239488 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.239665 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.240225 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.240233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.240237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.240657 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.240821 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.242206 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.242699 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260619 16:49:49.244635 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.176309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.0.0.1:50910)
0619 16:49:49.216630 (+ 40321us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.216662 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.218975 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.219632 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.219639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.220609 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.220978 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.220995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.222731 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.222752 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.224777 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.224794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.226365 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.226373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.226641 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.227435 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.227465 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.228756 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:49.238663 (+ 9907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.238671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.238677 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.239297 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.239853 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.239860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.239866 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.240083 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.240916 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:49.240924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:49.241235 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.243626 (+ 2391us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.244037 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40134,"thread_start_us":26054,"threads_started":1}
I20260619 16:49:49.245170 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.176820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.14.153.62:40135)
0619 16:49:49.177192 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.177200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.202322 (+ 25122us) server_negotiation.cc:407] Connection header received
0619 16:49:49.203663 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.203672 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.203783 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.203871 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.207134 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.208378 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.216898 (+ 8520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.217799 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.217924 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.229807 (+ 11883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.229814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.229818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.229862 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.237933 (+ 8071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.241821 (+ 3888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.241828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.241831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.242020 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.242667 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.242673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.242677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.243068 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.243244 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.244285 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.244513 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
I20260619 16:49:49.246392 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.175625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:55846)
0619 16:49:49.202172 (+ 26547us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.202201 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.202843 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.204146 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.204153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.205004 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.205437 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.205450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.208598 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.208603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.210311 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.210327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.218278 (+ 7951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.218287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.218564 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.224541 (+ 5977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.224561 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.225584 (+ 1023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:49.241043 (+ 15459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.241051 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.241061 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.241648 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.242193 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.242199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.242204 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.242500 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.243335 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:49.243341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:49.243910 (+ 569us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.245609 (+ 1699us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.245924 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26351,"spinlock_wait_cycles":30080,"thread_start_us":36936,"threads_started":1}
I20260619 16:49:49.250249 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.199547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46522 (local address 127.14.153.1:42227)
0619 16:49:49.220228 (+ 20681us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.220239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.220272 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:49:49.220340 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.220350 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.220670 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.222907 (+ 2237us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.223174 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.224735 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.234132 (+ 9397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.234961 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.235148 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.241104 (+ 5956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.241122 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.241142 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.241191 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.245420 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.246388 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.246397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.246402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.246572 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.247211 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.247220 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.247226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.247748 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.248069 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.249385 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.249699 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20480,"thread_start_us":21264,"threads_started":1}
I20260619 16:49:49.251796 21280 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 19 16:49:49 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887786, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.62@KRBTEST.COM for oryx/127.14.153.60@KRBTEST.COM
I20260619 16:49:49.272990 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.253222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:46829 (local address 127.0.0.1:58222)
0619 16:49:49.253390 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.253410 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.253518 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.254480 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.254488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.255113 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.255361 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.255376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.257065 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.257073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.258946 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.258965 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.260457 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.260466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.260676 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.261452 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.261483 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.264720 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:49.269656 (+ 4936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.269665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.269672 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.270293 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.270798 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.270805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.270812 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.270973 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.271747 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:49.271752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:49.271940 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.272105 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.272460 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:49:49.274694 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.253557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.14.153.60:46829)
0619 16:49:49.253776 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.253784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.253806 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:49:49.253944 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.253952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.254090 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.254218 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.255576 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.256893 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.259193 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.260038 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.260177 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.264936 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.264945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.264953 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.265002 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.269455 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.270481 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.270489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.270496 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.270658 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.271152 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.271159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.271164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.271604 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.273787 (+ 2183us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.273943 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.274257 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:49:49.276540 21096 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.153.60:46829: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a3a0303ea1b140869540f1fb602489b0"
member_type: NON_VOTER
last_known_addr {
host: "127.14.153.60"
port: 46829
}
attrs {
promote: true
}
}
dest_uuid: "d1416edc98814347a06ab1a14319c731"
cas_config_opid_index: 5
I20260619 16:49:49.277655 21096 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } }
I20260619 16:49:49.279449 21655 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a3a0303ea1b140869540f1fb602489b0 (127.14.153.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.283833 21704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 LEARNER]: Refusing update from remote peer d1416edc98814347a06ab1a14319c731: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:49:49.284901 21839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 16:49:49.285356 21786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:49.289110 21786 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 FOLLOWER]: Refusing update from remote peer d1416edc98814347a06ab1a14319c731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:49:49.289974 21839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:49.294840 21655 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.153.60:46829
I20260619 16:49:49.295461 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.295828 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:49.304225 21704 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a3a0303ea1b140869540f1fb602489b0 (127.14.153.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.309793 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.310087 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:49:49.319403 21658 master.cc:537] Master@127.14.153.60:46829 shutting down...
W20260619 16:49:49.319362 21765 tablet.cc:2369] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:49.339972 21785 raft_consensus.cc:858] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } }
W20260619 16:49:49.340194 21785 raft_consensus.cc:1565] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } }
I20260619 16:49:49.344188 21839 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: attempting to promote NON_VOTER eb37ccfbcfd841eb9a5e502f2bd87749 to VOTER
I20260619 16:49:49.345225 21839 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } }
I20260619 16:49:49.356330 21704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 LEARNER]: Refusing update from remote peer d1416edc98814347a06ab1a14319c731: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 16:49:49.357152 21839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 16:49:49.360258 21845 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, eb37ccfbcfd841eb9a5e502f2bd87749 (127.14.153.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.360998 21704 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, eb37ccfbcfd841eb9a5e502f2bd87749 (127.14.153.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.364984 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.365290 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:49.365545 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.365556 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.365751 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:49.365803 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:49:49.366329 21082 consensus_peers.cc:487] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 -> Peer a3a0303ea1b140869540f1fb602489b0 (127.14.153.60:46829): Couldn't send request to peer a3a0303ea1b140869540f1fb602489b0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:49:49.387045 21785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER eb37ccfbcfd841eb9a5e502f2bd87749 (127.14.153.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } }
I20260619 16:49:49.400271 21847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } }
I20260619 16:49:49.400658 21847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:49:49.401001 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1416edc98814347a06ab1a14319c731. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } }
I20260619 16:49:49.401203 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:49:49.402220 21850 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: NON_VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: true } } }
I20260619 16:49:49.402458 21850 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:49:49.404125 21658 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 16:49:49.404608 21658 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 16:49:49.404791 21658 tablet_replica.cc:330] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: stopping tablet replica
I20260619 16:49:49.412569 21658 master.cc:552] Master@127.14.153.60:46829 shutdown complete.
I20260619 16:49:49.412676 21658 master_runner.cc:305] Clearing existing system tablet
I20260619 16:49:49.413066 21658 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:49.413183 21658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:49.423692 21658 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.004s sys 0.008s
I20260619 16:49:49.438162 21852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:49.438781 21658 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:49.439312 21658 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.009s
I20260619 16:49:49.439486 21658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal
uuid: "a3a0303ea1b140869540f1fb602489b0"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "6a358974d9dddc5377c965040d295f76"
server_key_iv: "48323cbdf15f7fac9c595e9bd675f318"
server_key_version: "encryptionkey@0"
I20260619 16:49:49.442582 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 21157
I20260619 16:49:49.444932 21658 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:49.452489 21658 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:49:49.452790 21658 log.cc:1196] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 16:49:49.453435 21658 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Deleting consensus metadata
I20260619 16:49:49.453773 21658 master_runner.cc:315] Copying system tablet from 127.14.153.62:40135
I20260619 16:49:49.473560 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:42227
--local_ip_for_outbound_sockets=127.14.153.1
--tserver_master_addrs=127.14.153.62:40135
--webserver_port=41347
--webserver_interface=127.14.153.1
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab
--principal=oryx/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.153.62:40135,127.14.153.61:37409
--tserver_master_addrs=127.14.153.62:40135,127.14.153.61:37409,127.14.153.60:46829 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:49.498531 21658 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.153.62:40135
I20260619 16:49:49.528841 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.499523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.14.153.62:40135)
0619 16:49:49.499816 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.499826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.513460 (+ 13634us) server_negotiation.cc:407] Connection header received
0619 16:49:49.513923 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.513930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.514038 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.514111 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.515442 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.516571 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.518557 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.519266 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.519367 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.521941 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:49.521948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:49.521953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:49.521994 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:49.525678 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.526547 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.526553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.526556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.526705 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:49.527191 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:49.527196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:49.527198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:49.527557 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:49.527682 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.528251 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.528484 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 16:49:49.530124 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.499524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:55862)
0619 16:49:49.513304 (+ 13780us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.513338 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.513759 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.514437 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.514442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.514996 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.515267 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.515279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.516796 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.516801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.518382 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.518399 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.519643 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.519650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.519831 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.520545 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.520567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.521709 (+ 1142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:49.525911 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.525916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.525920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.526414 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.526849 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:49.526854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:49.526858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:49.527057 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:49.527778 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:49.527783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:49.527971 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.529370 (+ 1399us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.529688 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13608,"thread_start_us":13952,"threads_started":1}
I20260619 16:49:49.530934 21116 tablet_copy_service.cc:140] P d1416edc98814347a06ab1a14319c731: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a3a0303ea1b140869540f1fb602489b0 ({username='oryx', principal='oryx/127.14.153.60@KRBTEST.COM'} at 127.0.0.1:55862)
I20260619 16:49:49.531119 21116 tablet_copy_service.cc:161] P d1416edc98814347a06ab1a14319c731: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a3a0303ea1b140869540f1fb602489b0 at {username='oryx', principal='oryx/127.14.153.60@KRBTEST.COM'} at 127.0.0.1:55862: session id = a3a0303ea1b140869540f1fb602489b0-00000000000000000000000000000000
I20260619 16:49:49.535317 21116 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:49:49.537308 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:49:49.544255 21658 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: tablet copy: Starting download of 0 data blocks...
I20260619 16:49:49.544550 21658 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: tablet copy: Starting download of 1 WAL segments...
I20260619 16:49:49.546934 21658 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:49:49.551469 21116 tablet_copy_service.cc:342] P d1416edc98814347a06ab1a14319c731: Request end of tablet copy session a3a0303ea1b140869540f1fb602489b0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.153.60@KRBTEST.COM'} at 127.0.0.1:55862
I20260619 16:49:49.551616 21116 tablet_copy_service.cc:433] P d1416edc98814347a06ab1a14319c731: ending tablet copy session a3a0303ea1b140869540f1fb602489b0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a3a0303ea1b140869540f1fb602489b0
I20260619 16:49:49.562930 21658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:49.565800 21860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:49.568540 21861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:49.579339 21863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:49.579486 21658 server_base.cc:1034] running on GCE node
I20260619 16:49:49.579977 21658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:49.580605 21658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:49.581826 21658 hybrid_clock.cc:648] HybridClock initialized: now 1781887789581779 us; error 36 us; skew 500 ppm
I20260619 16:49:49.582077 21658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:49 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887789, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:49.584302 21658 init.cc:377] Logged in from keytab as oryx/127.14.153.60@KRBTEST.COM (short username oryx)
I20260619 16:49:49.587970 21658 webserver.cc:460] Webserver started at http://127.14.153.60:40585/ using document root <none> and password file <none>
I20260619 16:49:49.588331 21658 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:49.588440 21658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:49.591692 21658 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:49:49.593780 21870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:49.594220 21658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:49.594348 21658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal
uuid: "a3a0303ea1b140869540f1fb602489b0"
format_stamp: "Formatted at 2026-06-19 16:49:48 on dist-test-slave-qb36"
server_key: "6a358974d9dddc5377c965040d295f76"
server_key_iv: "48323cbdf15f7fac9c595e9bd675f318"
server_key_version: "encryptionkey@0"
I20260619 16:49:49.594542 21658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:49.613567 21658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:49.617838 21658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:49.642078 21658 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.60:46829
I20260619 16:49:49.642117 21921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.60:46829 every 8 connection(s)
I20260619 16:49:49.643143 21658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
I20260619 16:49:49.649961 21922 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:49:49.654167 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Bootstrap starting.
I20260619 16:49:49.670013 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:49:49.670635 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: Bootstrap complete.
I20260619 16:49:49.671838 21922 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } }
I20260619 16:49:49.672150 21922 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3a0303ea1b140869540f1fb602489b0, State: Initialized, Role: LEARNER
I20260619 16:49:49.672524 21922 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } }
I20260619 16:49:49.673380 21925 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.673907 21925 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:49:49.673782 21922 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:49.683857 21936 catalog_manager.cc:1259] Loaded cluster ID: 7ed1fb63941147198e32fd16bf7627d8
I20260619 16:49:49.683974 21936 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: loading cluster ID for follower catalog manager: success
I20260619 16:49:49.687808 21936 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: acquiring CA information for follower catalog manager: success
I20260619 16:49:49.690629 21936 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:49:49.775770 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.766660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:46829 (local address 127.0.0.1:58228)
0619 16:49:49.766971 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.766993 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.767147 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.768110 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.768119 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.768158 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:49.768359 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.768373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.770688 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.770694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.772669 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.772686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.774860 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.774868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.775097 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.775134 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:49:49.775223 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:49:49.775370 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260619 16:49:49.775883 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.766848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.14.153.60:46829)
0619 16:49:49.767684 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.767692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.767718 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:49:49.767805 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.767813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.767922 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.768512 (+ 590us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:49.768732 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.770518 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.772878 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.774664 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.774865 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.775183 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:49.775248 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 16:49:49.775456 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":505,"threads_started":1}
I20260619 16:49:49.777133 21897 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 16:49:49.777912 21925 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1416edc98814347a06ab1a14319c731. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:49.778220 21925 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: This master's current role is: LEARNER
W20260619 16:49:49.878703 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:49.879166 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:49.879338 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:49.887341 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:49.887456 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:49.887568 21856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:49.898665 21856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab
--principal=oryx/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:42227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=41347
--tserver_master_addrs=127.14.153.62:40135,127.14.153.61:37409,127.14.153.60:46829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:49.900031 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:49.901388 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:49.915995 21942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:49.916025 21945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:49.915987 21943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:49.916890 21856 server_base.cc:1034] running on GCE node
I20260619 16:49:49.917480 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:49.918390 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:49.919592 21856 hybrid_clock.cc:648] HybridClock initialized: now 1781887789919555 us; error 34 us; skew 500 ppm
I20260619 16:49:49.919935 21856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:49 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887789, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:49.927232 21856 init.cc:377] Logged in from keytab as oryx/127.14.153.1@KRBTEST.COM (short username oryx)
I20260619 16:49:49.928766 21856 webserver.cc:460] Webserver started at http://127.14.153.1:41347/ using document root <none> and password file <none>
I20260619 16:49:49.929464 21856 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:49.929605 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:49.935504 21856 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:49:49.938381 21952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:49.939916 21856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:49.940109 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "1494c019676b4562a447e90ea1f42977"
format_stamp: "Formatted at 2026-06-19 16:49:46 on dist-test-slave-qb36"
server_key: "a0eb9bc22b86d8c18c8ccc7ab80e3bca"
server_key_iv: "18f755663eb5e28fc9474fac5baba565"
server_key_version: "encryptionkey@0"
I20260619 16:49:49.941159 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:49.960947 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:49.967089 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:49.967391 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:49.968839 21856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:49.970788 21856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:49.970911 21856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:49.971019 21856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:49.971119 21856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 16:49:50.013116 21856 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:42227
I20260619 16:49:50.013170 22064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:42227 every 8 connection(s)
I20260619 16:49:50.014631 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:50.015200 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 21856
I20260619 16:49:50.015585 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 21291
Jun 19 16:49:50 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.2@KRBTEST.COM
Jun 19 16:49:50 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887789, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.60@KRBTEST.COM
Jun 19 16:49:50 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887789, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
Jun 19 16:49:50 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887789, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.61@KRBTEST.COM
I20260619 16:49:50.059659 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.022905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:46215 (local address 127.14.153.62:40135)
0619 16:49:50.023546 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.023555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.031692 (+ 8137us) server_negotiation.cc:407] Connection header received
0619 16:49:50.036879 (+ 5187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.036887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.036983 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.037057 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.040707 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.041787 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.044975 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.045634 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.045740 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.051376 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:50.051383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:50.051387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:50.051432 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:50.055259 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.056530 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.056537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.056542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.056698 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.057256 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.057264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.057267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.057653 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:50.057788 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:50.058992 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0619 16:49:50.059281 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479}
I20260619 16:49:50.059756 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.019887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:33885 (local address 127.14.153.60:46829)
0619 16:49:50.020129 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.020139 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.028240 (+ 8101us) server_negotiation.cc:407] Connection header received
0619 16:49:50.037993 (+ 9753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.037999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.038082 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.038175 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.040716 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.041836 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.044987 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.045697 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.045833 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.051267 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:50.051275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:50.051279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:50.051320 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:50.054898 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.056513 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.056518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.056521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.056723 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.057299 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.057304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.057308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.057687 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:50.057824 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:50.059063 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0619 16:49:50.059393 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:49:50.060554 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.023147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:51499 (local address 127.14.153.61:37409)
0619 16:49:50.035537 (+ 12390us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.035545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.037440 (+ 1895us) server_negotiation.cc:407] Connection header received
0619 16:49:50.037640 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.037645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.037734 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.037849 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.040714 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.041836 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.044962 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.045606 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.045713 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.051984 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:50.051992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:50.051998 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:50.052054 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:50.055880 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.056735 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.056743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.056749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.056906 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.057337 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.057345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.057351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.057669 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:50.057797 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:50.059542 (+ 1745us) server_negotiation.cc:299] Negotiation successful
0619 16:49:50.060200 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12237,"thread_start_us":12115,"threads_started":1}
I20260619 16:49:50.062049 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.017030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.14.153.2:44093)
0619 16:49:50.034967 (+ 17937us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.034979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.035019 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:49:50.035348 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.035361 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.035779 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.035938 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.038687 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.040362 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.042286 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.043207 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.043334 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.052970 (+ 9636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:50.052991 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:50.053014 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:50.053069 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:50.057894 (+ 4825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.058923 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.058934 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.058940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.059125 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.059565 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.059570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.059574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.060045 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:50.060315 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:50.061427 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0619 16:49:50.061679 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16628,"thread_start_us":17477,"threads_started":1}
I20260619 16:49:50.062245 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.023003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.14.153.1:51499)
0619 16:49:50.037325 (+ 14322us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:50.037339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:50.037491 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:50.038045 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:50.038049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:50.040189 (+ 2140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:50.040559 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.040568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.042023 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.042028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.044807 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.044816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.046076 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.046083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.046644 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.047417 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:50.047459 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:50.051751 (+ 4292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:50.056048 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.056053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.056058 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.056606 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.057048 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.057053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.057056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.057220 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.057895 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:50.057904 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:50.059270 (+ 1366us) client_negotiation.cc:769] Sending connection context
0619 16:49:50.061103 (+ 1833us) client_negotiation.cc:240] Negotiation successful
0619 16:49:50.061535 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14221,"mutex_wait_us":528,"spinlock_wait_cycles":8064,"thread_start_us":14552,"threads_started":1}
I20260619 16:49:50.062319 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.022804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.1:46215)
0619 16:49:50.031556 (+ 8752us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:50.031581 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:50.036527 (+ 4946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:50.037692 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:50.037709 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:50.039400 (+ 1691us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:50.040497 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.040527 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.042011 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.042016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.044789 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.044804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.046159 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.046166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.046595 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.047424 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:50.047455 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:50.051119 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:50.055434 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.055441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.055448 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.056381 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.056840 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.056844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.056847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.057131 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.057908 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:50.057911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:50.058438 (+ 527us) client_negotiation.cc:769] Sending connection context
0619 16:49:50.059971 (+ 1533us) client_negotiation.cc:240] Negotiation successful
0619 16:49:50.060576 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8581,"mutex_wait_us":576,"spinlock_wait_cycles":1408,"thread_start_us":8417,"threads_started":1}
I20260619 16:49:50.062345 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.019199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:46829 (local address 127.14.153.1:33885)
0619 16:49:50.027982 (+ 8783us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:50.028065 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:50.037809 (+ 9744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:50.038381 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:50.038387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:50.039390 (+ 1003us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:50.040495 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.040529 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.042023 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.042028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.044817 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.044831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.046251 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.046260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.046618 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.047627 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:50.047653 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:50.050942 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:50.055097 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.055110 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.055132 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.056368 (+ 1236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.056874 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.056882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.056886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.057154 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.058341 (+ 1187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:50.058348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:50.058628 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 16:49:50.060046 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0619 16:49:50.060756 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8325,"mutex_wait_us":671,"spinlock_wait_cycles":19584,"thread_start_us":8025,"threads_started":1}
I20260619 16:49:50.063761 21414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:50.064009 22066 heartbeater.cc:347] Connected to a master server at 127.14.153.62:40135
I20260619 16:49:50.064124 22067 heartbeater.cc:347] Connected to a master server at 127.14.153.61:37409
I20260619 16:49:50.064328 22067 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:50.065199 22067 heartbeater.cc:511] Master 127.14.153.61:37409 requested a full tablet report, sending...
I20260619 16:49:50.066066 22065 heartbeater.cc:347] Connected to a master server at 127.14.153.60:46829
I20260619 16:49:50.066192 22065 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:50.066663 22065 heartbeater.cc:511] Master 127.14.153.60:46829 requested a full tablet report, sending...
I20260619 16:49:50.064334 22066 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:50.067718 21694 ts_manager.cc:194] Registered new tserver with Master: 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227)
I20260619 16:49:50.067817 22066 heartbeater.cc:511] Master 127.14.153.62:40135 requested a full tablet report, sending...
I20260619 16:49:50.068470 21887 ts_manager.cc:194] Registered new tserver with Master: 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227)
I20260619 16:49:50.068905 21096 ts_manager.cc:194] Re-registered known tserver with Master: 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227)
I20260619 16:49:50.069882 21096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:46215
I20260619 16:49:50.148609 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 21291
I20260619 16:49:50.165017 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:44093
--local_ip_for_outbound_sockets=127.14.153.2
--tserver_master_addrs=127.14.153.62:40135
--webserver_port=41147
--webserver_interface=127.14.153.2
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab
--principal=oryx/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.153.62:40135,127.14.153.61:37409
--tserver_master_addrs=127.14.153.62:40135,127.14.153.61:37409,127.14.153.60:46829 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:50.286525 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: NON_VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: true } } }
I20260619 16:49:50.286875 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:50.288717 22077 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: attempting to promote NON_VOTER a3a0303ea1b140869540f1fb602489b0 to VOTER
I20260619 16:49:50.289444 22077 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } }
I20260619 16:49:50.291445 21897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 LEARNER]: Refusing update from remote peer d1416edc98814347a06ab1a14319c731: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 16:49:50.291545 21704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Refusing update from remote peer d1416edc98814347a06ab1a14319c731: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 16:49:50.292104 22078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:49:50.292563 22077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:49:50.294902 22078 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3a0303ea1b140869540f1fb602489b0 (127.14.153.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:50.295660 21704 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3a0303ea1b140869540f1fb602489b0 (127.14.153.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:50.299263 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:50.299549 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:50.299801 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:50.300026 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:50.300973 22081 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:50.301534 22081 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:50.304538 22077 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:50.304800 22077 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:50.305147 21897 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3a0303ea1b140869540f1fb602489b0 (127.14.153.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:50.308171 22082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1416edc98814347a06ab1a14319c731" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:50.308457 22082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:49:50.505661 22076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:50.505971 22076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:50.506070 22076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:50.513450 22076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:50.513561 22076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:50.513674 22076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:50.523361 22076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab
--principal=oryx/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:44093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=41147
--tserver_master_addrs=127.14.153.62:40135,127.14.153.61:37409,127.14.153.60:46829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:50.524695 22076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:50.526044 22076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:50.539912 22088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:50.539968 22087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:50.539939 22090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:50.540572 22076 server_base.cc:1034] running on GCE node
I20260619 16:49:50.541252 22076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:50.542197 22076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:50.543426 22076 hybrid_clock.cc:648] HybridClock initialized: now 1781887790543376 us; error 62 us; skew 500 ppm
I20260619 16:49:50.543754 22076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:50 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887790, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:50.547259 22076 init.cc:377] Logged in from keytab as oryx/127.14.153.2@KRBTEST.COM (short username oryx)
I20260619 16:49:50.548702 22076 webserver.cc:460] Webserver started at http://127.14.153.2:41147/ using document root <none> and password file <none>
I20260619 16:49:50.549400 22076 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:50.549561 22076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:50.555019 22076 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:50.557950 22097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:50.559360 22076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:50.559533 22076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "f65afa21f9194948ab94de36d244d1dd"
format_stamp: "Formatted at 2026-06-19 16:49:47 on dist-test-slave-qb36"
server_key: "78e317658a696516524be53cf429d7ad"
server_key_iv: "bb2c12011960e99ff05e869cb25be902"
server_key_version: "encryptionkey@0"
I20260619 16:49:50.560534 22076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:50.591663 22076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:50.596268 22076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:50.596596 22076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:50.597635 22076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:50.599080 22076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:50.599207 22076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:50.599334 22076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:50.599432 22076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:50.632026 22076 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:44093
I20260619 16:49:50.632119 22209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:44093 every 8 connection(s)
I20260619 16:49:50.633594 22076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:50.638484 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 22076
I20260619 16:49:50.638815 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 21424
Jun 19 16:49:50 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887790, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
Jun 19 16:49:50 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.3@KRBTEST.COM
Jun 19 16:49:50 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887790, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.60@KRBTEST.COM
Jun 19 16:49:50 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887790, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.61@KRBTEST.COM
I20260619 16:49:50.689082 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.639915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.14.153.3:46809)
0619 16:49:50.662682 (+ 22767us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.662692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.662724 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:49:50.662800 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.662808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.663981 (+ 1173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.664151 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.666905 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.668489 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.670882 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.671752 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.671864 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.679745 (+ 7881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:50.679757 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:50.679798 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:50.679865 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:50.684689 (+ 4824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.685594 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.685603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.685607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.685764 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.686444 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.686451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.686455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.686957 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:50.687275 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:50.687899 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 16:49:50.688201 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22576,"thread_start_us":23158,"threads_started":1}
I20260619 16:49:50.689143 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.636878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:53195 (local address 127.14.153.62:40135)
0619 16:49:50.648305 (+ 11427us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.648312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.663868 (+ 15556us) server_negotiation.cc:407] Connection header received
0619 16:49:50.666222 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.666228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.666318 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.666387 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.669176 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.670306 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.673170 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.673829 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.673912 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.679389 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:50.679398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:50.679403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:50.679449 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:50.683048 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.684304 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.684310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.684315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.684477 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.685585 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.685591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.685596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.685975 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:50.686124 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:50.688575 (+ 2451us) server_negotiation.cc:299] Negotiation successful
0619 16:49:50.688787 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11270,"thread_start_us":11429,"threads_started":1}
I20260619 16:49:50.689986 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.637514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.2:53195)
0619 16:49:50.663350 (+ 25836us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:50.663377 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:50.666094 (+ 2717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:50.666804 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:50.666816 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:50.668184 (+ 1368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:50.668938 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.668965 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.670486 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.670491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.673010 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.673024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.674217 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.674224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.674595 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.675741 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:50.675764 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:50.679111 (+ 3347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:50.683592 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.683603 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.683624 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.684160 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.685216 (+ 1056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.685222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.685227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.685461 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.687149 (+ 1688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:50.687157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:50.687633 (+ 476us) client_negotiation.cc:769] Sending connection context
0619 16:49:50.687958 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 16:49:50.688511 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25649,"mutex_wait_us":3968,"spinlock_wait_cycles":16768,"thread_start_us":20359,"threads_started":1}
I20260619 16:49:50.690428 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.636733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:56743 (local address 127.14.153.61:37409)
0619 16:49:50.664733 (+ 28000us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.664739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.664757 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:49:50.667083 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.667089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.667180 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.667254 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.669149 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.670280 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.673169 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.673799 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.673898 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.682708 (+ 8810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:50.682716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:50.682735 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:50.682831 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:50.686521 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.687322 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.687328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.687331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.687470 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.687923 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.687931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.687934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.688281 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:50.688428 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:50.689844 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0619 16:49:50.690090 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27883,"thread_start_us":27789,"threads_started":1}
I20260619 16:49:50.691351 21547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:50.691560 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.637706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:44773 (local address 127.14.153.60:46829)
0619 16:49:50.664233 (+ 26527us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.664243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.664268 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:49:50.666273 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.666280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.666380 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.666480 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.669531 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.670678 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.673209 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.674013 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.674144 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.682796 (+ 8652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:50.682805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:50.682812 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:50.682856 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:50.686943 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.687810 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.687816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.687819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.687950 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:50.688369 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:50.688374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:50.688378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:50.689085 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:50.689696 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:50.690797 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0619 16:49:50.691017 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26337,"thread_start_us":26938,"threads_started":1}
I20260619 16:49:50.692251 22211 heartbeater.cc:347] Connected to a master server at 127.14.153.62:40135
I20260619 16:49:50.692593 22211 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:50.693320 22211 heartbeater.cc:511] Master 127.14.153.62:40135 requested a full tablet report, sending...
I20260619 16:49:50.694872 21096 ts_manager.cc:194] Re-registered known tserver with Master: f65afa21f9194948ab94de36d244d1dd (127.14.153.2:44093)
I20260619 16:49:50.695266 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.637921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:46829 (local address 127.14.153.2:44773)
0619 16:49:50.662508 (+ 24587us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:50.662535 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:50.665847 (+ 3312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:50.666809 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:50.666816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:50.667886 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:50.668939 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.668966 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.670851 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.670857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.673039 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.673055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.674441 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.674448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.674743 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.676028 (+ 1285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:50.676054 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:50.680538 (+ 4484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:50.687112 (+ 6574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.687116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.687120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.687660 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.688079 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.688084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.688087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.688241 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.689774 (+ 1533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:50.689779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:50.689920 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 16:49:50.690382 (+ 462us) client_negotiation.cc:240] Negotiation successful
0619 16:49:50.694655 (+ 4273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23623,"mutex_wait_us":3218,"thread_start_us":25596,"threads_started":1}
I20260619 16:49:50.695333 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.637514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.14.153.2:56743)
0619 16:49:50.658611 (+ 21097us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:50.658720 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:50.665842 (+ 7122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:50.667469 (+ 1627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:50.667473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:50.668103 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:50.668938 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.668965 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.670485 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.670491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.673010 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.673024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.674217 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.674224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.674592 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.675639 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:50.675680 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:50.682415 (+ 6735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:50.686662 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.686668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.686672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.687180 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.687615 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:50.687619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:50.687623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:50.687792 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:50.689386 (+ 1594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:50.689391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:50.689564 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 16:49:50.690662 (+ 1098us) client_negotiation.cc:240] Negotiation successful
0619 16:49:50.694512 (+ 3850us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20743,"mutex_wait_us":244,"spinlock_wait_cycles":7040,"thread_start_us":20462,"threads_started":1}
I20260619 16:49:50.696117 21096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:53195
I20260619 16:49:50.696504 22210 heartbeater.cc:347] Connected to a master server at 127.14.153.60:46829
I20260619 16:49:50.696623 22210 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:50.697117 22210 heartbeater.cc:511] Master 127.14.153.60:46829 requested a full tablet report, sending...
I20260619 16:49:50.697172 22212 heartbeater.cc:347] Connected to a master server at 127.14.153.61:37409
I20260619 16:49:50.697280 22212 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:50.697789 22212 heartbeater.cc:511] Master 127.14.153.61:37409 requested a full tablet report, sending...
I20260619 16:49:50.698375 21887 ts_manager.cc:194] Registered new tserver with Master: f65afa21f9194948ab94de36d244d1dd (127.14.153.2:44093)
I20260619 16:49:50.698809 21694 ts_manager.cc:194] Registered new tserver with Master: f65afa21f9194948ab94de36d244d1dd (127.14.153.2:44093)
I20260619 16:49:50.787945 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 21424
I20260619 16:49:50.806428 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:46809
--local_ip_for_outbound_sockets=127.14.153.3
--tserver_master_addrs=127.14.153.62:40135
--webserver_port=44145
--webserver_interface=127.14.153.3
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab
--principal=oryx/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.153.62:40135,127.14.153.61:37409
--tserver_master_addrs=127.14.153.62:40135,127.14.153.61:37409,127.14.153.60:46829 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:51.072163 22066 heartbeater.cc:503] Master 127.14.153.62:40135 was elected leader, sending a full tablet report...
W20260619 16:49:51.132212 22223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:51.132543 22223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:51.132701 22223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:51.140477 22223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:51.140641 22223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:51.140842 22223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:51.150910 22223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab
--principal=oryx/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:46809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=44145
--tserver_master_addrs=127.14.153.62:40135,127.14.153.61:37409,127.14.153.60:46829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:51.152356 22223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:51.153692 22223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:51.167232 22229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:51.167342 22228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:51.167339 22231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:51.168715 22223 server_base.cc:1034] running on GCE node
I20260619 16:49:51.169322 22223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:51.170261 22223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:51.171458 22223 hybrid_clock.cc:648] HybridClock initialized: now 1781887791171408 us; error 49 us; skew 500 ppm
I20260619 16:49:51.171741 22223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:51 dist-test-slave-qb36 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887791, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:51.175542 22223 init.cc:377] Logged in from keytab as oryx/127.14.153.3@KRBTEST.COM (short username oryx)
I20260619 16:49:51.177301 22223 webserver.cc:460] Webserver started at http://127.14.153.3:44145/ using document root <none> and password file <none>
I20260619 16:49:51.178052 22223 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:51.178203 22223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:51.183868 22223 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:49:51.186754 22238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:51.188267 22223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:49:51.188454 22223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "d6c6e5f441b74d4a9c9b889a030dc609"
format_stamp: "Formatted at 2026-06-19 16:49:47 on dist-test-slave-qb36"
server_key: "e2bf2c97ef66c4c7a4f6d35a41a48867"
server_key_iv: "bb6c2d08077d7a0eec6548a92102e7bb"
server_key_version: "encryptionkey@0"
I20260619 16:49:51.189455 22223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:51.214489 22223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:51.219440 22223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:51.219739 22223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:51.220700 22223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:51.222105 22223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:51.222225 22223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:51.222353 22223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:51.222440 22223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:51.254055 22223 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:46809
I20260619 16:49:51.254149 22350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:46809 every 8 connection(s)
I20260619 16:49:51.255642 22223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:51.257344 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 22223
Jun 19 16:49:51 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887791, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
Jun 19 16:49:51 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887791, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.61@KRBTEST.COM
Jun 19 16:49:51 dist-test-slave-qb36 krb5kdc[21051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887791, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.60@KRBTEST.COM
I20260619 16:49:51.324551 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.258755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:50765 (local address 127.14.153.61:37409)
0619 16:49:51.297009 (+ 38254us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.297019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.297043 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:49:51.297117 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.297125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.297224 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.299100 (+ 1876us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.299251 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.300562 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.303964 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.306947 (+ 2983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.307975 (+ 1028us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.311705 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.311711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.311716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.311762 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.315688 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.317880 (+ 2192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.317888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.317895 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.318121 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.319480 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.319488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.319493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.319834 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.319995 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.323654 (+ 3659us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.323956 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38072,"thread_start_us":39511,"threads_started":1}
I20260619 16:49:51.326567 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.260375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.14.153.3:50765)
0619 16:49:51.292241 (+ 31866us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.292269 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.292437 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.297459 (+ 5022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.297467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.298278 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:51.298837 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.298855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.301430 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.301443 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.303714 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.303737 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.307246 (+ 3509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.307256 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.307545 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.308265 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:51.308296 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:51.311486 (+ 3190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:51.316816 (+ 5330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:51.316829 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:51.316852 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:51.317661 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:51.318293 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:51.318301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:51.318307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:51.318574 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:51.320090 (+ 1516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:51.320103 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:51.320729 (+ 626us) client_negotiation.cc:769] Sending connection context
0619 16:49:51.321110 (+ 381us) client_negotiation.cc:240] Negotiation successful
0619 16:49:51.321868 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31678,"thread_start_us":26760,"threads_started":1}
I20260619 16:49:51.327186 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.272849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.3:59057)
0619 16:49:51.293433 (+ 20584us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.293458 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.293570 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.294041 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.294045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.294845 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:51.296382 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.296419 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.298578 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.298591 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.301768 (+ 3177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.301799 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.303638 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.303648 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.304084 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.305377 (+ 1293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:51.305403 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:51.308062 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:51.319425 (+ 11363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:51.319433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:51.319437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:51.319938 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:51.323508 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:51.323513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:51.323516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:51.323673 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:51.325640 (+ 1967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:51.325645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:51.325902 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 16:49:51.326039 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 16:49:51.326303 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20432,"spinlock_wait_cycles":24448,"thread_start_us":19679,"threads_started":1}
I20260619 16:49:51.328212 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.272857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:59057 (local address 127.14.153.62:40135)
0619 16:49:51.291454 (+ 18597us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.291463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.293526 (+ 2063us) server_negotiation.cc:407] Connection header received
0619 16:49:51.293675 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.293683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.293799 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.293940 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.296596 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.298139 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.302050 (+ 3911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.303015 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.303185 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.308370 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.308377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.308382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.308420 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.319303 (+ 10883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.323146 (+ 3843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.323156 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.323162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.323386 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.325013 (+ 1627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.325022 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.325027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.325511 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.325660 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.327555 (+ 1895us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.327806 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18418,"thread_start_us":18609,"threads_started":1}
I20260619 16:49:51.329489 22353 heartbeater.cc:347] Connected to a master server at 127.14.153.61:37409
I20260619 16:49:51.329849 22353 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:51.330790 22353 heartbeater.cc:511] Master 127.14.153.61:37409 requested a full tablet report, sending...
I20260619 16:49:51.333050 22352 heartbeater.cc:347] Connected to a master server at 127.14.153.62:40135
I20260619 16:49:51.333191 22352 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:51.337563 21694 ts_manager.cc:194] Registered new tserver with Master: d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809)
I20260619 16:49:51.338649 22352 heartbeater.cc:511] Master 127.14.153.62:40135 requested a full tablet report, sending...
I20260619 16:49:51.340557 21096 ts_manager.cc:194] Re-registered known tserver with Master: d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809)
I20260619 16:49:51.341790 21096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:59057
I20260619 16:49:51.344750 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.259549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:46829 (local address 127.14.153.3:50235)
0619 16:49:51.284484 (+ 24935us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.284555 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.289381 (+ 4826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.293779 (+ 4398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.293794 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.300795 (+ 7001us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:51.301181 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.301197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.307013 (+ 5816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.307019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.309715 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.309729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.311874 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.311886 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.312187 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.312688 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:51.312713 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:51.325779 (+ 13066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:51.339470 (+ 13691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:51.339474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:51.339478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:51.340052 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:51.342922 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:51.342928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:51.342931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:51.343157 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:51.343788 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:51.343793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:51.343925 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 16:49:51.344038 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:49:51.344382 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24470,"thread_start_us":33392,"threads_started":1}
I20260619 16:49:51.345750 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.260228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:50235 (local address 127.14.153.60:46829)
0619 16:49:51.290867 (+ 30639us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.290877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.290909 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:49:51.290995 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.291003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.291270 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.291460 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.304989 (+ 13529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.306445 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.310005 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.310703 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.310845 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.327447 (+ 16602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.327454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.327459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.327499 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.336313 (+ 8814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.342599 (+ 6286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.342605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.342608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.342783 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.343304 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.343309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.343312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.343639 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.345179 (+ 1540us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.345283 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.345525 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30436,"thread_start_us":32620,"threads_started":1}
I20260619 16:49:51.346556 22351 heartbeater.cc:347] Connected to a master server at 127.14.153.60:46829
I20260619 16:49:51.346689 22351 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:51.347126 22351 heartbeater.cc:511] Master 127.14.153.60:46829 requested a full tablet report, sending...
I20260619 16:49:51.348362 21887 ts_manager.cc:194] Registered new tserver with Master: d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809)
I20260619 16:49:51.349673 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.312215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.14.153.62:40135)
0619 16:49:51.318598 (+ 6383us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.318606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.331312 (+ 12706us) server_negotiation.cc:407] Connection header received
0619 16:49:51.331373 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.331377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.331457 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.331564 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.334144 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.335285 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.338304 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.339065 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.339153 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.342128 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.342137 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.342145 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.342185 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.346496 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.347475 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.347483 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.347489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.347618 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.348042 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.348047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.348051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.348299 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.348590 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.349027 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.349250 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6254,"thread_start_us":4409,"threads_started":1}
I20260619 16:49:51.352229 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.314968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.14.153.61:37409)
0619 16:49:51.324903 (+ 9935us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.324911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.330907 (+ 5996us) server_negotiation.cc:407] Connection header received
0619 16:49:51.331106 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.331111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.331196 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.331303 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.333712 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.334876 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.337376 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.338046 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.338143 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.342357 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.342364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.342370 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.342409 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.346424 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.349974 (+ 3550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.349980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.349986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.350146 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.350600 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.350608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.350613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.350982 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.351280 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.351608 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.351855 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9819,"thread_start_us":14315,"threads_started":1}
I20260619 16:49:51.358228 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.313040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.14.153.60:46829)
0619 16:49:51.330387 (+ 17347us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.330393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.330412 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:49:51.330466 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.330473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.330539 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.330645 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.333651 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.335308 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.338337 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.338985 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.339141 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.349528 (+ 10387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.349533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.349536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.349563 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.353157 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.355749 (+ 2592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.355753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.355755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.355891 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.356306 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.356310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.356312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.356602 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.356804 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.357236 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.357426 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17240,"thread_start_us":17145,"threads_started":1}
I20260619 16:49:51.353843 21096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:51.362457 21096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:51.401856 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.385090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.14.153.1:42227)
0619 16:49:51.394547 (+ 9457us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.394555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.394591 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:49:51.394673 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.394679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.395114 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.395420 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:51.396070 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.397416 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.399542 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.400826 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.400971 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.401199 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.401430 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.401582 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9297,"thread_start_us":9199,"threads_started":1}
I20260619 16:49:51.401863 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.384669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:42227 (local address 127.0.0.1:46536)
0619 16:49:51.385329 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.385356 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.385521 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.395397 (+ 9876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.395402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.395428 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:51.395783 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.395792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.397549 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.397555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.399389 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.399403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.401044 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.401051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.401238 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.401271 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:49:51.401383 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:49:51.401511 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":9333,"threads_started":1}
I20260619 16:49:51.403997 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.385361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:44093 (local address 127.0.0.1:50842)
0619 16:49:51.395383 (+ 10022us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.395406 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.395568 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.396477 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.396482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.396509 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:51.396840 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.396850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.399019 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.399053 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.401071 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.401086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.403116 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.403122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.403333 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.403368 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:49:51.403461 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:49:51.403587 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9819,"mutex_wait_us":2,"thread_start_us":8387,"threads_started":1}
I20260619 16:49:51.404729 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.385578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.14.153.2:44093)
0619 16:49:51.386474 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.386487 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.395751 (+ 9264us) server_negotiation.cc:407] Connection header received
0619 16:49:51.395910 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.395922 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.396236 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.396500 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:51.397763 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.398894 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.401256 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.402630 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.402757 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.402935 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.404302 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.404485 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":292,"threads_started":1}
I20260619 16:49:51.406332 22000 tablet_service.cc:1467] Processing CreateTablet for tablet 9dfd4bcbfe974acc9e246747963c7f22 (DEFAULT_TABLE table=test-table [id=26d67da9d0944ca0a9a405f7da4f0a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:51.407642 22000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dfd4bcbfe974acc9e246747963c7f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:51.408792 22145 tablet_service.cc:1467] Processing CreateTablet for tablet 9dfd4bcbfe974acc9e246747963c7f22 (DEFAULT_TABLE table=test-table [id=26d67da9d0944ca0a9a405f7da4f0a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:51.409605 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.384669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:46809 (local address 127.0.0.1:41116)
0619 16:49:51.393459 (+ 8790us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.393489 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.393649 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.399821 (+ 6172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.399826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.399860 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:51.400319 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.400331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.402229 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.402234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.404153 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.404170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.408784 (+ 4614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.408791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.408975 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.409013 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:49:51.409085 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:49:51.409234 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8609,"mutex_wait_us":67,"thread_start_us":324,"threads_started":1}
I20260619 16:49:51.410132 22145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dfd4bcbfe974acc9e246747963c7f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:51.410713 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.384977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.14.153.3:46809)
0619 16:49:51.399255 (+ 14278us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.399266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.399300 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:49:51.399391 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.399400 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.399625 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.399829 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:51.400520 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.402113 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.407057 (+ 4944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.408536 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.409967 (+ 1431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.410183 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.410253 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.410412 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14083,"thread_start_us":12332,"threads_started":1}
I20260619 16:49:51.414316 22286 tablet_service.cc:1467] Processing CreateTablet for tablet 9dfd4bcbfe974acc9e246747963c7f22 (DEFAULT_TABLE table=test-table [id=26d67da9d0944ca0a9a405f7da4f0a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:51.415659 22286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dfd4bcbfe974acc9e246747963c7f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:51.422653 22388 tablet_bootstrap.cc:492] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: Bootstrap starting.
I20260619 16:49:51.424171 22389 tablet_bootstrap.cc:492] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: Bootstrap starting.
I20260619 16:49:51.426204 22388 tablet_bootstrap.cc:654] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:51.427327 22388 log.cc:824] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:51.427942 22389 tablet_bootstrap.cc:654] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:51.429080 22389 log.cc:824] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:51.430222 22388 tablet_bootstrap.cc:492] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: No bootstrap required, opened a new log
I20260619 16:49:51.430485 22388 ts_tablet_manager.cc:1397] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:49:51.434912 22392 tablet_bootstrap.cc:492] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: Bootstrap starting.
I20260619 16:49:51.438397 22388 raft_consensus.cc:348] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } }
I20260619 16:49:51.438784 22388 raft_consensus.cc:374] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:51.438874 22388 raft_consensus.cc:729] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1494c019676b4562a447e90ea1f42977, State: Initialized, Role: FOLLOWER
I20260619 16:49:51.439935 22392 tablet_bootstrap.cc:654] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:51.439635 22388 consensus_queue.cc:260] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } }
I20260619 16:49:51.441532 22392 log.cc:824] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:51.444985 22389 tablet_bootstrap.cc:492] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: No bootstrap required, opened a new log
I20260619 16:49:51.445408 22389 ts_tablet_manager.cc:1397] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.015s
I20260619 16:49:51.445961 22392 tablet_bootstrap.cc:492] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: No bootstrap required, opened a new log
I20260619 16:49:51.446208 22392 ts_tablet_manager.cc:1397] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260619 16:49:51.447078 22388 ts_tablet_manager.cc:1428] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260619 16:49:51.455384 22392 raft_consensus.cc:348] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } }
I20260619 16:49:51.455755 22392 raft_consensus.cc:374] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:51.455850 22392 raft_consensus.cc:729] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c6e5f441b74d4a9c9b889a030dc609, State: Initialized, Role: FOLLOWER
I20260619 16:49:51.456478 22392 consensus_queue.cc:260] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } }
I20260619 16:49:51.457558 22352 heartbeater.cc:503] Master 127.14.153.62:40135 was elected leader, sending a full tablet report...
I20260619 16:49:51.457412 22389 raft_consensus.cc:348] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } }
I20260619 16:49:51.457868 22389 raft_consensus.cc:374] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:51.457995 22389 raft_consensus.cc:729] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65afa21f9194948ab94de36d244d1dd, State: Initialized, Role: FOLLOWER
I20260619 16:49:51.458598 22389 consensus_queue.cc:260] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } }
I20260619 16:49:51.459932 22211 heartbeater.cc:503] Master 127.14.153.62:40135 was elected leader, sending a full tablet report...
I20260619 16:49:51.460636 22389 ts_tablet_manager.cc:1428] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: Time spent starting tablet: real 0.015s user 0.008s sys 0.002s
I20260619 16:49:51.461476 22392 ts_tablet_manager.cc:1428] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260619 16:49:51.484130 22396 raft_consensus.cc:482] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:51.484449 22396 raft_consensus.cc:504] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } }
I20260619 16:49:51.486178 22396 leader_election.cc:288] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809), 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227)
I20260619 16:49:51.493599 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.486755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:60585 (local address 127.14.153.1:42227)
0619 16:49:51.486986 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.486995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.487744 (+ 749us) server_negotiation.cc:407] Connection header received
0619 16:49:51.487947 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.487951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.488007 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.488114 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:51.488736 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.489596 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.491716 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.492679 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.492851 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.492927 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.493201 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.493322 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:49:51.493888 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.486493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:55113 (local address 127.14.153.3:46809)
0619 16:49:51.486707 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.486714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.487709 (+ 995us) server_negotiation.cc:407] Connection header received
0619 16:49:51.487932 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.487936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.487992 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.488062 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:51.488698 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.489416 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.491662 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.492945 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.493132 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.493205 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.493521 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.493644 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:49:51.494168 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.486612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:42227 (local address 127.14.153.2:60585)
0619 16:49:51.487608 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.487622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.487794 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.488147 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.488152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.488174 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:51.488565 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.488573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.489763 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.489767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.491541 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.491552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.492875 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.492881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.493034 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.493058 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:49:51.493177 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:49:51.493309 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":15,"spinlock_wait_cycles":8576,"thread_start_us":536,"threads_started":1}
I20260619 16:49:51.494202 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.486364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:46809 (local address 127.14.153.2:55113)
0619 16:49:51.487564 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.487592 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.487773 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.488119 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.488123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.488148 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:51.488560 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.488570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.489581 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.489586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.491513 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.491525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.493158 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.493164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.493372 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.493395 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:49:51.493497 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:49:51.493629 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":40,"spinlock_wait_cycles":18688,"thread_start_us":1023,"threads_started":1}
I20260619 16:49:51.495286 22020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfd4bcbfe974acc9e246747963c7f22" candidate_uuid: "f65afa21f9194948ab94de36d244d1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1494c019676b4562a447e90ea1f42977" is_pre_election: true
I20260619 16:49:51.495342 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfd4bcbfe974acc9e246747963c7f22" candidate_uuid: "f65afa21f9194948ab94de36d244d1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" is_pre_election: true
I20260619 16:49:51.495837 22306 raft_consensus.cc:2454] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65afa21f9194948ab94de36d244d1dd in term 0.
I20260619 16:49:51.495839 22020 raft_consensus.cc:2454] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65afa21f9194948ab94de36d244d1dd in term 0.
I20260619 16:49:51.496555 22099 leader_election.cc:302] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1494c019676b4562a447e90ea1f42977, d6c6e5f441b74d4a9c9b889a030dc609, f65afa21f9194948ab94de36d244d1dd; no voters:
I20260619 16:49:51.496981 22396 raft_consensus.cc:2781] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:51.497115 22396 raft_consensus.cc:482] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:51.497218 22396 raft_consensus.cc:3037] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:51.499444 22396 raft_consensus.cc:504] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } }
I20260619 16:49:51.500020 22396 leader_election.cc:288] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [CANDIDATE]: Term 1 election: Requested vote from peers d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809), 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227)
I20260619 16:49:51.500384 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfd4bcbfe974acc9e246747963c7f22" candidate_uuid: "f65afa21f9194948ab94de36d244d1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c6e5f441b74d4a9c9b889a030dc609"
I20260619 16:49:51.500490 22020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfd4bcbfe974acc9e246747963c7f22" candidate_uuid: "f65afa21f9194948ab94de36d244d1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1494c019676b4562a447e90ea1f42977"
I20260619 16:49:51.500581 22306 raft_consensus.cc:3037] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:51.500710 22020 raft_consensus.cc:3037] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:51.502825 22020 raft_consensus.cc:2454] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65afa21f9194948ab94de36d244d1dd in term 1.
I20260619 16:49:51.503194 22101 leader_election.cc:302] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1494c019676b4562a447e90ea1f42977, f65afa21f9194948ab94de36d244d1dd; no voters:
I20260619 16:49:51.503377 22306 raft_consensus.cc:2454] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65afa21f9194948ab94de36d244d1dd in term 1.
I20260619 16:49:51.503489 22396 raft_consensus.cc:2781] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:51.503968 22396 raft_consensus.cc:686] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 1 LEADER]: Becoming Leader. State: Replica: f65afa21f9194948ab94de36d244d1dd, State: Running, Role: LEADER
I20260619 16:49:51.504477 22396 consensus_queue.cc:237] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } }
I20260619 16:49:51.509338 21095 catalog_manager.cc:5495] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd reported cstate change: term changed from 0 to 1, leader changed from <none> to f65afa21f9194948ab94de36d244d1dd (127.14.153.2). New cstate: current_term: 1 leader_uuid: "f65afa21f9194948ab94de36d244d1dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } health_report { overall_health: UNKNOWN } } }
W20260619 16:49:51.537494 22074 tablet.cc:2369] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:51.555598 22354 tablet.cc:2369] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:51.557940 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.552329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.14.153.2:44093)
0619 16:49:51.552541 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.552548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.552568 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:49:51.552719 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.552723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.552816 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.552980 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:51.553353 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.554078 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.555640 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.556107 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.556246 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.556770 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:51.557307 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:51.557472 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.557527 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.557621 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:49:51.566625 22306 raft_consensus.cc:1261] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [term 1 FOLLOWER]: Refusing update from remote peer f65afa21f9194948ab94de36d244d1dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:51.566622 22020 raft_consensus.cc:1261] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [term 1 FOLLOWER]: Refusing update from remote peer f65afa21f9194948ab94de36d244d1dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:51.567699 22396 consensus_queue.cc:1035] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:51.567993 22399 consensus_queue.cc:1035] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:51.583618 22402 mvcc.cc:205] Tried to move back new op lower bound from 7298612394246971392 to 7298612394006151168. Current Snapshot: MvccSnapshot[applied={T|T < 7298612394246971392}]
I20260619 16:49:51.603785 21096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:51.604081 21096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:51.610021 21096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1416edc98814347a06ab1a14319c731: Sending DeleteTablet for 3 replicas of tablet 9dfd4bcbfe974acc9e246747963c7f22
I20260619 16:49:51.611382 22000 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dfd4bcbfe974acc9e246747963c7f22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:51 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:46536
I20260619 16:49:51.611479 22286 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dfd4bcbfe974acc9e246747963c7f22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:51 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:41116
I20260619 16:49:51.611641 22145 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dfd4bcbfe974acc9e246747963c7f22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:51 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:50842
I20260619 16:49:51.627097 22414 tablet_replica.cc:330] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: stopping tablet replica
I20260619 16:49:51.628825 22414 raft_consensus.cc:2229] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:51.629189 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 21064
I20260619 16:49:51.630414 21126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:51.630409 22414 raft_consensus.cc:2258] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:51.630656 22413 tablet_replica.cc:330] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: stopping tablet replica
I20260619 16:49:51.631223 22413 raft_consensus.cc:2229] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:51.631801 22413 raft_consensus.cc:2258] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:51.633574 22414 ts_tablet_manager.cc:1905] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:51.635668 22412 tablet_replica.cc:330] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: stopping tablet replica
I20260619 16:49:51.639986 22413 ts_tablet_manager.cc:1905] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:51.641381 22414 ts_tablet_manager.cc:1918] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:51.641593 22414 log.cc:1196] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/9dfd4bcbfe974acc9e246747963c7f22
I20260619 16:49:51.642238 22414 ts_tablet_manager.cc:1939] T 9dfd4bcbfe974acc9e246747963c7f22 P f65afa21f9194948ab94de36d244d1dd: Deleting consensus metadata
I20260619 16:49:51.651589 22412 raft_consensus.cc:2229] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:51.652323 22412 raft_consensus.cc:2258] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:51.659945 22412 ts_tablet_manager.cc:1905] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:51.667138 22412 ts_tablet_manager.cc:1918] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:51.667333 22412 log.cc:1196] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/9dfd4bcbfe974acc9e246747963c7f22
I20260619 16:49:51.667309 22413 ts_tablet_manager.cc:1918] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:51.667490 22413 log.cc:1196] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/9dfd4bcbfe974acc9e246747963c7f22
I20260619 16:49:51.667910 22412 ts_tablet_manager.cc:1939] T 9dfd4bcbfe974acc9e246747963c7f22 P 1494c019676b4562a447e90ea1f42977: Deleting consensus metadata
I20260619 16:49:51.668114 22413 ts_tablet_manager.cc:1939] T 9dfd4bcbfe974acc9e246747963c7f22 P d6c6e5f441b74d4a9c9b889a030dc609: Deleting consensus metadata
I20260619 16:49:51.670130 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.616164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.14.153.61:37409)
0619 16:49:51.616332 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.616339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.627535 (+ 11196us) server_negotiation.cc:407] Connection header received
0619 16:49:51.627838 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.627844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.627931 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.628007 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.636038 (+ 8031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.637247 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.657388 (+ 20141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.658135 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.658291 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.662610 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.662617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.662623 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.662667 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.666320 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.667373 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.667379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.667445 (+ 66us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.667675 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.668313 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.668318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.668321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.668703 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.668940 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.669541 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.669773 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:49:51.687103 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.627764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.14.153.60:46829)
0619 16:49:51.628042 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.628051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.673015 (+ 44964us) server_negotiation.cc:407] Connection header received
0619 16:49:51.673225 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.673231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.673340 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.673405 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.675310 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.676168 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.677576 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.678113 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.678250 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.680833 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.680839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.680844 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.680886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.684087 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.684731 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.684742 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.684745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.684865 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.685266 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.685271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.685273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.685539 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.685793 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.686657 (+ 864us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.686826 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:49:51.832163 21083 catalog_manager.cc:4841] TS 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227): tablet 9dfd4bcbfe974acc9e246747963c7f22 (table test-table [id=26d67da9d0944ca0a9a405f7da4f0a73]) successfully deleted
I20260619 16:49:51.832792 21081 catalog_manager.cc:4841] TS f65afa21f9194948ab94de36d244d1dd (127.14.153.2:44093): tablet 9dfd4bcbfe974acc9e246747963c7f22 (table test-table [id=26d67da9d0944ca0a9a405f7da4f0a73]) successfully deleted
W20260619 16:49:51.832969 21081 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.14.153.2:44093, user_credentials={real_user=slave}} blocked reactor thread for 188447us
I20260619 16:49:51.833473 21081 catalog_manager.cc:4841] TS d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809): tablet 9dfd4bcbfe974acc9e246747963c7f22 (table test-table [id=26d67da9d0944ca0a9a405f7da4f0a73]) successfully deleted
I20260619 16:49:51.833523 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 21064
I20260619 16:49:51.888238 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.865121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.14.153.60:46829)
0619 16:49:51.865302 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.865307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.869832 (+ 4525us) server_negotiation.cc:407] Connection header received
0619 16:49:51.870423 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.870433 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.870545 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.870639 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.874497 (+ 3858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.875932 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.877792 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.878490 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.878616 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.882182 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.882189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.882194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.882235 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.885312 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.885964 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.885977 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.885979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.886095 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.886479 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.886484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.886487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.886897 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.887182 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.887587 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.887841 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:49:51.889634 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.864647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.14.153.61:37409)
0619 16:49:51.865003 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.865012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.873457 (+ 8445us) server_negotiation.cc:407] Connection header received
0619 16:49:51.873760 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.873767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.873851 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.873920 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.876494 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.877285 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.879359 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.879987 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.880117 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.883120 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:51.883127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:51.883134 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:51.883173 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:51.886482 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.887284 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.887290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.887293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.887480 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:51.887931 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:51.887938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:51.887943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:51.888307 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:51.888536 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:51.889083 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 16:49:51.889314 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260619 16:49:52.589387 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:52.588112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.2:53823)
0619 16:49:52.589025 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:52.589174 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":679,"threads_started":1}
W20260619 16:49:52.589861 22211 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:40135 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
I20260619 16:49:52.595774 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:52.594850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.1:48081)
0619 16:49:52.595409 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:52.595571 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":347,"threads_started":1}
W20260619 16:49:52.596263 22066 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:40135 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
I20260619 16:49:52.598443 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:52.597713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.3:41359)
0619 16:49:52.598167 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:52.598275 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":248,"threads_started":1}
W20260619 16:49:52.598853 22352 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:40135 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
I20260619 16:49:53.110816 22427 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1416edc98814347a06ab1a14319c731)
I20260619 16:49:53.111159 22427 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } }
I20260619 16:49:53.111584 22428 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1416edc98814347a06ab1a14319c731)
I20260619 16:49:53.111855 22428 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } }
I20260619 16:49:53.112761 22427 leader_election.cc:288] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1416edc98814347a06ab1a14319c731 (127.14.153.62:40135), a3a0303ea1b140869540f1fb602489b0 (127.14.153.60:46829)
I20260619 16:49:53.115850 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.113240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35402)
0619 16:49:53.115265 (+ 2025us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.115499 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1925,"thread_start_us":1787,"threads_started":1}
W20260619 16:49:53.119926 21871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
I20260619 16:49:53.121718 22428 leader_election.cc:288] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1416edc98814347a06ab1a14319c731 (127.14.153.62:40135), eb37ccfbcfd841eb9a5e502f2bd87749 (127.14.153.61:37409)
I20260619 16:49:53.127207 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.112972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35398)
0619 16:49:53.126653 (+ 13681us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.126864 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13541,"mutex_wait_us":62,"threads_started":1}
I20260619 16:49:53.128569 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.126087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35412)
0619 16:49:53.128116 (+ 2029us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.128325 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1910}
W20260619 16:49:53.128469 21678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
W20260619 16:49:53.128822 21871 leader_election.cc:334] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1416edc98814347a06ab1a14319c731 (127.14.153.62:40135): Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
I20260619 16:49:53.129159 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.112972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:46829 (local address 127.0.0.1:38968)
0619 16:49:53.113641 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.113662 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:53.113797 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:53.120076 (+ 6279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:53.120081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:53.120111 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:53.120459 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.120472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.123535 (+ 3063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.123545 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.126112 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.126130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.128072 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.128081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.128343 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.128396 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 16:49:53.128492 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:49:53.128649 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":3,"thread_start_us":375,"threads_started":1}
I20260619 16:49:53.130053 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.115278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.14.153.60:46829)
0619 16:49:53.118378 (+ 3100us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.118384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.118402 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:49:53.118461 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.118466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.118543 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.118635 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:53.120671 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.122340 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.126370 (+ 4030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.127730 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.127865 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.127979 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:53.129561 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0619 16:49:53.129738 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3001,"thread_start_us":4323,"threads_started":1}
I20260619 16:49:53.130723 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a3a0303ea1b140869540f1fb602489b0" is_pre_election: true
I20260619 16:49:53.131191 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.130759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35416)
0619 16:49:53.130940 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.131020 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 16:49:53.131150 21897 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb37ccfbcfd841eb9a5e502f2bd87749 in term 1.
W20260619 16:49:53.131475 21678 leader_election.cc:334] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1416edc98814347a06ab1a14319c731 (127.14.153.62:40135): Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
I20260619 16:49:53.131771 21680 leader_election.cc:302] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3a0303ea1b140869540f1fb602489b0, eb37ccfbcfd841eb9a5e502f2bd87749; no voters: d1416edc98814347a06ab1a14319c731
I20260619 16:49:53.132145 22427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:49:53.132282 22427 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1416edc98814347a06ab1a14319c731)
I20260619 16:49:53.132375 22427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:49:53.134933 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.113936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:37409 (local address 127.0.0.1:39404)
0619 16:49:53.117263 (+ 3327us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.117286 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:53.117443 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:53.128422 (+ 10979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:53.128432 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:53.128467 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:53.128695 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.128705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.130539 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.130546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.132722 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.132736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.134212 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.134219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.134395 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.134428 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:49:53.134523 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:49:53.134673 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3204,"thread_start_us":13923,"threads_started":1}
I20260619 16:49:53.135058 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.114065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.14.153.61:37409)
0619 16:49:53.127856 (+ 13791us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.127862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.127878 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:49:53.127936 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.127941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.128026 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.128122 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:53.128891 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.130331 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.132951 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.134016 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.134144 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.134277 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:53.134565 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 16:49:53.134762 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":464,"server-negotiator.queue_time_us":13659,"thread_start_us":13443,"threads_started":1}
I20260619 16:49:53.135281 22427 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } }
I20260619 16:49:53.135692 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3a0303ea1b140869540f1fb602489b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" is_pre_election: true
I20260619 16:49:53.136083 21704 raft_consensus.cc:2379] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3a0303ea1b140869540f1fb602489b0 in current term 2: Already voted for candidate eb37ccfbcfd841eb9a5e502f2bd87749 in this term.
I20260619 16:49:53.136533 22427 leader_election.cc:288] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [CANDIDATE]: Term 2 election: Requested vote from peers d1416edc98814347a06ab1a14319c731 (127.14.153.62:40135), a3a0303ea1b140869540f1fb602489b0 (127.14.153.60:46829)
I20260619 16:49:53.136620 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.136218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35428)
0619 16:49:53.136379 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.136464 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:49:53.136619 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a3a0303ea1b140869540f1fb602489b0"
I20260619 16:49:53.136750 21872 leader_election.cc:302] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3a0303ea1b140869540f1fb602489b0; no voters: d1416edc98814347a06ab1a14319c731, eb37ccfbcfd841eb9a5e502f2bd87749
I20260619 16:49:53.136888 21897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:49:53.137975 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.137608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35432)
0619 16:49:53.137755 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.137844 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:49:53.138127 21678 leader_election.cc:334] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d1416edc98814347a06ab1a14319c731 (127.14.153.62:40135): Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
I20260619 16:49:53.139600 21897 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb37ccfbcfd841eb9a5e502f2bd87749 in term 2.
I20260619 16:49:53.139840 22428 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:49:53.140069 21680 leader_election.cc:302] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3a0303ea1b140869540f1fb602489b0, eb37ccfbcfd841eb9a5e502f2bd87749; no voters: d1416edc98814347a06ab1a14319c731
I20260619 16:49:53.140350 22427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:49:53.140883 22427 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [term 2 LEADER]: Becoming Leader. State: Replica: eb37ccfbcfd841eb9a5e502f2bd87749, State: Running, Role: LEADER
I20260619 16:49:53.141285 22427 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } }
I20260619 16:49:53.143285 22437 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb37ccfbcfd841eb9a5e502f2bd87749. Latest consensus state: current_term: 2 leader_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:53.143831 22437 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:53.144479 22439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:53.148128 22439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:53.148933 22439 catalog_manager.cc:1259] Loaded cluster ID: 7ed1fb63941147198e32fd16bf7627d8
I20260619 16:49:53.149040 22439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:53.150154 22439 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:53.150918 22439 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Loaded TSK: 0
I20260619 16:49:53.151769 22439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:53.189493 21693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:53.196820 21693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:53.201079 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.200581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35438)
0619 16:49:53.200799 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.200895 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 16:49:53.201045 21897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [term 2 FOLLOWER]: Refusing update from remote peer eb37ccfbcfd841eb9a5e502f2bd87749: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 16:49:53.201797 22437 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 16:49:53.202055 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.201696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35442)
0619 16:49:53.201844 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.201921 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 16:49:53.202204 21678 consensus_peers.cc:487] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 -> Peer d1416edc98814347a06ab1a14319c731 (127.14.153.62:40135): Couldn't send request to peer d1416edc98814347a06ab1a14319c731. Status: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:49:53.204249 22428 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb37ccfbcfd841eb9a5e502f2bd87749. Latest consensus state: current_term: 2 leader_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:53.204506 22428 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:53.206117 22427 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:53.206467 22427 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:53.206565 22428 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d1416edc98814347a06ab1a14319c731" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40135 } } peers { permanent_uuid: "eb37ccfbcfd841eb9a5e502f2bd87749" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 37409 } attrs { promote: false } } peers { permanent_uuid: "a3a0303ea1b140869540f1fb602489b0" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 46829 } attrs { promote: false } } }
I20260619 16:49:53.206840 22428 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a0303ea1b140869540f1fb602489b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:53.207301 22441 mvcc.cc:205] Tried to move back new op lower bound from 7298612400944488448 to 7298612400709971968. Current Snapshot: MvccSnapshot[applied={T|T < 7298612400944488448 or (T in {7298612400944488448})}]
I20260619 16:49:53.232779 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.215625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:42227 (local address 127.0.0.1:52094)
0619 16:49:53.215821 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.215834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:53.215956 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:53.216328 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:53.216333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:53.216360 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:53.216697 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.216709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.226898 (+ 10189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.226908 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.230291 (+ 3383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.230311 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.231978 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.231985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.232172 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.232208 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:49:53.232298 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:49:53.232437 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 16:49:53.233081 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.215300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.14.153.1:42227)
0619 16:49:53.215737 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.215743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.215911 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:49:53.216087 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.216092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.216165 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.222875 (+ 6710us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:53.223201 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.224679 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.230515 (+ 5836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.231760 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.231894 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.232007 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:53.232474 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 16:49:53.232739 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":960,"threads_started":1}
I20260619 16:49:53.233866 22000 tablet_service.cc:1467] Processing CreateTablet for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 (DEFAULT_TABLE table=test-table [id=6945d5b2dbfc47378a31c014c83df097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:53.234334 22000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:53.235584 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.214733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:44093 (local address 127.0.0.1:49290)
0619 16:49:53.215056 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.215078 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:53.215180 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:53.227854 (+ 12674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:53.227861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:53.227891 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:53.228132 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.228143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.229094 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.229121 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.231744 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.231756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.234634 (+ 2878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.234640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.234974 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.235010 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:49:53.235135 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 16:49:53.235270 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":43}
I20260619 16:49:53.235597 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.214911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.14.153.2:44093)
0619 16:49:53.227526 (+ 12615us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.227531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.227548 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:49:53.227608 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.227612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.227670 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.227813 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:53.228297 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.228957 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.233186 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.234473 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.234637 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.234773 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:53.235162 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 16:49:53.235311 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12507,"thread_start_us":12995,"threads_started":1}
I20260619 16:49:53.237305 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.217151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.14.153.3:46809)
0619 16:49:53.229873 (+ 12722us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.229882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.230635 (+ 753us) server_negotiation.cc:407] Connection header received
0619 16:49:53.230825 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.230830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.230912 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.231020 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:53.231590 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.232591 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.235088 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.236320 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.236458 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.236572 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:53.236860 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 16:49:53.236989 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12539,"thread_start_us":12735,"threads_started":1}
I20260619 16:49:53.237357 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.217028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:46809 (local address 127.0.0.1:53412)
0619 16:49:53.230516 (+ 13488us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.230532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:53.230688 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:53.231067 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:53.231074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:53.231100 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:53.231443 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.231450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.232741 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.232751 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.234907 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.234927 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.236525 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.236531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.236702 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.236725 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:49:53.236836 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:49:53.236965 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13369,"mutex_wait_us":45,"thread_start_us":13187,"threads_started":1}
I20260619 16:49:53.238160 22286 tablet_service.cc:1467] Processing CreateTablet for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 (DEFAULT_TABLE table=test-table [id=6945d5b2dbfc47378a31c014c83df097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:53.238605 22286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:53.239981 22145 tablet_service.cc:1467] Processing CreateTablet for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 (DEFAULT_TABLE table=test-table [id=6945d5b2dbfc47378a31c014c83df097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:53.240456 22145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:53.240779 22450 tablet_bootstrap.cc:492] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: Bootstrap starting.
I20260619 16:49:53.242563 22450 tablet_bootstrap.cc:654] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:53.244917 22450 tablet_bootstrap.cc:492] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: No bootstrap required, opened a new log
I20260619 16:49:53.244918 22452 tablet_bootstrap.cc:492] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: Bootstrap starting.
I20260619 16:49:53.245050 22450 ts_tablet_manager.cc:1397] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260619 16:49:53.245698 22450 raft_consensus.cc:348] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } }
I20260619 16:49:53.245954 22450 raft_consensus.cc:374] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:53.246052 22450 raft_consensus.cc:729] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1494c019676b4562a447e90ea1f42977, State: Initialized, Role: FOLLOWER
I20260619 16:49:53.246327 22450 consensus_queue.cc:260] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } }
I20260619 16:49:53.246744 22452 tablet_bootstrap.cc:654] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:53.253321 22450 ts_tablet_manager.cc:1428] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260619 16:49:53.255859 22067 heartbeater.cc:503] Master 127.14.153.61:37409 was elected leader, sending a full tablet report...
I20260619 16:49:53.255939 22456 tablet_bootstrap.cc:492] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: Bootstrap starting.
I20260619 16:49:53.258507 22456 tablet_bootstrap.cc:654] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:53.259827 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.247399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.1:50625)
0619 16:49:53.259244 (+ 11845us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.259464 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11649,"thread_start_us":11449,"threads_started":1}
I20260619 16:49:53.261332 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.260644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.1:55007)
0619 16:49:53.261114 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.261195 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"threads_started":1}
I20260619 16:49:53.261508 22452 tablet_bootstrap.cc:492] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: No bootstrap required, opened a new log
I20260619 16:49:53.261724 22452 ts_tablet_manager.cc:1397] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260619 16:49:53.262694 22452 raft_consensus.cc:348] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } }
I20260619 16:49:53.262981 22452 raft_consensus.cc:374] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:53.263123 22452 raft_consensus.cc:729] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c6e5f441b74d4a9c9b889a030dc609, State: Initialized, Role: FOLLOWER
I20260619 16:49:53.263448 22452 consensus_queue.cc:260] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } }
I20260619 16:49:53.264353 22452 ts_tablet_manager.cc:1428] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 16:49:53.270284 22456 tablet_bootstrap.cc:492] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: No bootstrap required, opened a new log
I20260619 16:49:53.270473 22456 ts_tablet_manager.cc:1397] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260619 16:49:53.271265 22353 heartbeater.cc:503] Master 127.14.153.61:37409 was elected leader, sending a full tablet report...
I20260619 16:49:53.271252 22456 raft_consensus.cc:348] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } }
I20260619 16:49:53.271490 22456 raft_consensus.cc:374] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:53.271610 22456 raft_consensus.cc:729] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65afa21f9194948ab94de36d244d1dd, State: Initialized, Role: FOLLOWER
I20260619 16:49:53.271867 22456 consensus_queue.cc:260] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } }
I20260619 16:49:53.274684 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.265555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.3:37795)
0619 16:49:53.274187 (+ 8632us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.274348 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8463,"threads_started":1}
I20260619 16:49:53.275724 22456 ts_tablet_manager.cc:1428] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260619 16:49:53.277122 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.276509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.3:50177)
0619 16:49:53.276878 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.276964 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259}
I20260619 16:49:53.277935 22212 heartbeater.cc:503] Master 127.14.153.61:37409 was elected leader, sending a full tablet report...
I20260619 16:49:53.279945 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.275948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.2:53353)
0619 16:49:53.279513 (+ 3565us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.279654 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3427,"thread_start_us":3290,"threads_started":1}
I20260619 16:49:53.280862 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.280493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.2:41755)
0619 16:49:53.280681 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.280763 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 16:49:53.415930 22213 tablet.cc:2369] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:53.574023 22461 raft_consensus.cc:482] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:53.574335 22461 raft_consensus.cc:504] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } }
I20260619 16:49:53.575091 22461 leader_election.cc:288] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227), d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809)
I20260619 16:49:53.575745 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d92f8eb19ec4ccb9b4af500e4fccf83" candidate_uuid: "f65afa21f9194948ab94de36d244d1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" is_pre_election: true
I20260619 16:49:53.575745 22020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d92f8eb19ec4ccb9b4af500e4fccf83" candidate_uuid: "f65afa21f9194948ab94de36d244d1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1494c019676b4562a447e90ea1f42977" is_pre_election: true
I20260619 16:49:53.576025 22306 raft_consensus.cc:2454] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65afa21f9194948ab94de36d244d1dd in term 0.
I20260619 16:49:53.576042 22020 raft_consensus.cc:2454] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65afa21f9194948ab94de36d244d1dd in term 0.
I20260619 16:49:53.576565 22099 leader_election.cc:302] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6c6e5f441b74d4a9c9b889a030dc609, f65afa21f9194948ab94de36d244d1dd; no voters:
I20260619 16:49:53.576853 22461 raft_consensus.cc:2781] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:53.576968 22461 raft_consensus.cc:482] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:53.577059 22461 raft_consensus.cc:3037] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:53.579350 22461 raft_consensus.cc:504] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } }
I20260619 16:49:53.579972 22461 leader_election.cc:288] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [CANDIDATE]: Term 1 election: Requested vote from peers 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227), d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809)
I20260619 16:49:53.580595 22020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d92f8eb19ec4ccb9b4af500e4fccf83" candidate_uuid: "f65afa21f9194948ab94de36d244d1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1494c019676b4562a447e90ea1f42977"
I20260619 16:49:53.580595 22306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d92f8eb19ec4ccb9b4af500e4fccf83" candidate_uuid: "f65afa21f9194948ab94de36d244d1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c6e5f441b74d4a9c9b889a030dc609"
I20260619 16:49:53.580845 22306 raft_consensus.cc:3037] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:53.580858 22020 raft_consensus.cc:3037] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:53.583876 22020 raft_consensus.cc:2454] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65afa21f9194948ab94de36d244d1dd in term 1.
I20260619 16:49:53.583876 22306 raft_consensus.cc:2454] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65afa21f9194948ab94de36d244d1dd in term 1.
I20260619 16:49:53.584348 22101 leader_election.cc:302] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1494c019676b4562a447e90ea1f42977, d6c6e5f441b74d4a9c9b889a030dc609, f65afa21f9194948ab94de36d244d1dd; no voters:
I20260619 16:49:53.584620 22461 raft_consensus.cc:2781] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:53.585016 22461 raft_consensus.cc:686] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 1 LEADER]: Becoming Leader. State: Replica: f65afa21f9194948ab94de36d244d1dd, State: Running, Role: LEADER
I20260619 16:49:53.585346 22461 consensus_queue.cc:237] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } }
I20260619 16:49:53.586102 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.585661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.2:56919)
0619 16:49:53.585843 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.585947 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:49:53.588423 21694 catalog_manager.cc:5495] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd reported cstate change: term changed from 0 to 1, leader changed from <none> to f65afa21f9194948ab94de36d244d1dd (127.14.153.2). New cstate: current_term: 1 leader_uuid: "f65afa21f9194948ab94de36d244d1dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65afa21f9194948ab94de36d244d1dd" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:53.614969 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.610212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.14.153.2:44093)
0619 16:49:53.610431 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.610438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.610457 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:49:53.610679 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.610687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.610766 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.610869 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:53.611283 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.611980 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.613303 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.613710 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.613842 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.614249 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:53.614362 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:53.614462 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:53.614605 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 16:49:53.614687 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:49:53.618096 22306 raft_consensus.cc:1261] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [term 1 FOLLOWER]: Refusing update from remote peer f65afa21f9194948ab94de36d244d1dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:53.618109 22020 raft_consensus.cc:1261] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [term 1 FOLLOWER]: Refusing update from remote peer f65afa21f9194948ab94de36d244d1dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:53.618803 22463 consensus_queue.cc:1035] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c6e5f441b74d4a9c9b889a030dc609" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:53.619161 22461 consensus_queue.cc:1035] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1494c019676b4562a447e90ea1f42977" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 42227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:53.621783 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.621312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.3:55161)
0619 16:49:53.621469 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.621570 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 16:49:53.621832 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.621070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.1:51407)
0619 16:49:53.621362 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.621556 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260619 16:49:53.627195 22467 mvcc.cc:205] Tried to move back new op lower bound from 7298612402653630464 to 7298612402527883264. Current Snapshot: MvccSnapshot[applied={T|T < 7298612402653630464}]
I20260619 16:49:53.628795 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.628341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.1:52725)
0619 16:49:53.628551 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.628643 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 16:49:53.628895 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.628494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.2:57441)
0619 16:49:53.628678 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.628755 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 16:49:53.631850 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.631321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.3:53053)
0619 16:49:53.631553 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.631659 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 16:49:53.636049 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.635017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.14.153.2:38165)
0619 16:49:53.635767 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.635869 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 16:49:53.639341 21694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:53.639583 21694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:53.645674 21694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb37ccfbcfd841eb9a5e502f2bd87749: Sending DeleteTablet for 3 replicas of tablet 1d92f8eb19ec4ccb9b4af500e4fccf83
I20260619 16:49:53.647171 22145 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:53 UTC) from {username='oryx', principal='oryx/127.14.153.61@KRBTEST.COM'} at 127.0.0.1:49290
I20260619 16:49:53.648025 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 21856
I20260619 16:49:53.647976 22286 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:53 UTC) from {username='oryx', principal='oryx/127.14.153.61@KRBTEST.COM'} at 127.0.0.1:53412
I20260619 16:49:53.648401 22285 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:53 UTC) from {username='oryx', principal='oryx/127.14.153.61@KRBTEST.COM'} at 127.0.0.1:53412
I20260619 16:49:53.648828 22475 tablet_replica.cc:330] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: stopping tablet replica
I20260619 16:49:53.649096 22475 raft_consensus.cc:2229] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:53.649523 22475 raft_consensus.cc:2258] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:53.650056 22000 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:53 UTC) from {username='oryx', principal='oryx/127.14.153.61@KRBTEST.COM'} at 127.0.0.1:52094
I20260619 16:49:53.650563 22475 ts_tablet_manager.cc:1905] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:53.650823 22478 tablet_replica.cc:330] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: stopping tablet replica
I20260619 16:49:53.651118 22478 raft_consensus.cc:2229] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:53.651502 22478 raft_consensus.cc:2258] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:53.652717 22478 ts_tablet_manager.cc:1905] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:53.657879 22475 ts_tablet_manager.cc:1918] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:53.658032 22475 log.cc:1196] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/1d92f8eb19ec4ccb9b4af500e4fccf83
I20260619 16:49:53.658493 22475 ts_tablet_manager.cc:1939] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P f65afa21f9194948ab94de36d244d1dd: Deleting consensus metadata
I20260619 16:49:53.659776 21679 catalog_manager.cc:4841] TS f65afa21f9194948ab94de36d244d1dd (127.14.153.2:44093): tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 (table test-table [id=6945d5b2dbfc47378a31c014c83df097]) successfully deleted
I20260619 16:49:53.661267 22478 ts_tablet_manager.cc:1918] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:53.661502 22478 log.cc:1196] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/1d92f8eb19ec4ccb9b4af500e4fccf83
I20260619 16:49:53.662190 22478 ts_tablet_manager.cc:1939] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P 1494c019676b4562a447e90ea1f42977: Deleting consensus metadata
I20260619 16:49:53.663632 21681 catalog_manager.cc:4841] TS 1494c019676b4562a447e90ea1f42977 (127.14.153.1:42227): tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 (table test-table [id=6945d5b2dbfc47378a31c014c83df097]) successfully deleted
I20260619 16:49:53.671553 22476 tablet_replica.cc:330] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: stopping tablet replica
I20260619 16:49:53.671800 22476 raft_consensus.cc:2229] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:53.672151 22476 raft_consensus.cc:2258] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:53.673343 22476 ts_tablet_manager.cc:1905] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:53.679522 22476 ts_tablet_manager.cc:1918] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:53.679663 22476 log.cc:1196] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/1d92f8eb19ec4ccb9b4af500e4fccf83
I20260619 16:49:53.680137 22476 ts_tablet_manager.cc:1939] T 1d92f8eb19ec4ccb9b4af500e4fccf83 P d6c6e5f441b74d4a9c9b889a030dc609: Deleting consensus metadata
I20260619 16:49:53.681416 21679 catalog_manager.cc:4841] TS d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809): tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 (table test-table [id=6945d5b2dbfc47378a31c014c83df097]) successfully deleted
W20260619 16:49:53.681864 21679 catalog_manager.cc:4808] TS d6c6e5f441b74d4a9c9b889a030dc609 (127.14.153.3:46809): delete failed for tablet 1d92f8eb19ec4ccb9b4af500e4fccf83 because the tablet was not found. No further retry: Not found: Tablet not found: 1d92f8eb19ec4ccb9b4af500e4fccf83
I20260619 16:49:53.687085 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.649272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.14.153.1:42227)
0619 16:49:53.649487 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.649494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.672307 (+ 22813us) server_negotiation.cc:407] Connection header received
0619 16:49:53.672645 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.672652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.672734 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.672845 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:53.674719 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.675823 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.677610 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.678261 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.678379 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.681727 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:53.681742 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:53.681757 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:53.681802 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:53.684713 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:53.685339 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:53.685346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:53.685349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:53.685474 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:53.685784 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:53.685787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:53.685789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:53.686157 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:53.686368 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:53.686592 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 16:49:53.686799 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:49:53.687845 22060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:53.731484 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.731014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35458)
0619 16:49:53.731226 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.731339 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 16:49:53.732988 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.732588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:40135 (local address 127.0.0.1:35466)
0619 16:49:53.732734 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.732870 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:40135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:49:53.800102 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 21856
I20260619 16:49:53.817895 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 22076
I20260619 16:49:53.831528 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.818647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.14.153.2:44093)
0619 16:49:53.818866 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.818871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.818952 (+ 81us) server_negotiation.cc:407] Connection header received
0619 16:49:53.819164 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.819169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.819236 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.819320 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:53.820524 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.821236 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.822388 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.822829 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.822948 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.825679 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:53.825694 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:53.825711 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:53.825743 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:53.828486 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:53.829378 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:53.829389 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:53.829395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:53.829561 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:53.830024 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:53.830028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:53.830031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:53.830394 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:53.830622 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:53.830894 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 16:49:53.831277 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:49:53.832341 22205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:53.951018 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 22076
I20260619 16:49:53.973012 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 22223
I20260619 16:49:53.986471 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.973743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.14.153.3:46809)
0619 16:49:53.974248 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.974253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.974270 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:49:53.974323 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.974327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.974386 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.974547 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:53.976131 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.976806 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.978024 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.978430 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.978568 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.980998 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:53.981022 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:53.981040 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:53.981078 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:53.983814 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:53.984456 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:53.984470 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:53.984472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:53.984592 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:53.984981 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:53.984992 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:53.984994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:53.985321 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:53.985641 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:53.985906 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 16:49:53.986128 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
I20260619 16:49:53.987221 22346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:54.080417 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 22223
I20260619 16:49:54.097690 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 21557
I20260619 16:49:54.098749 21724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:54.220425 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 21557
W20260619 16:49:54.241219 21871 connection.cc:535] server connection from 127.0.0.1:38968 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:49:54.241575 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 21658
I20260619 16:49:54.242717 21917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:49:54.356698 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 21658
2026-06-19T16:49:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9866 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:54 dist-test-slave-qb36 krb5kdc[22488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:54 dist-test-slave-qb36 krb5kdc[22488](info): set up 2 sockets
Jun 19 16:49:54 dist-test-slave-qb36 krb5kdc[22488](info): commencing operation
krb5kdc: starting...
W20260619 16:49:55.460035 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:49:55 dist-test-slave-qb36 krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.62.keytab.
Entry for principal oryx/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.62.keytab.
I20260619 16:49:55.678783 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:36287
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.62.keytab
--principal=oryx/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:36287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:56.013221 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:56.013604 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:56.013780 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:56.021358 22501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:56.021526 22501 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:49:56.021632 22501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:56.021763 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:56.021852 22501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:56.021942 22501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:56.032326 22501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:36287
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.62.keytab
--principal=oryx/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:36287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:56.033733 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:56.035128 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:56.047785 22506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:56.047879 22507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:56.047823 22509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:56.048583 22501 server_base.cc:1034] running on GCE node
I20260619 16:49:56.049382 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:56.050949 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:56.052244 22501 hybrid_clock.cc:648] HybridClock initialized: now 1781887796052193 us; error 38 us; skew 500 ppm
I20260619 16:49:56.052614 22501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:56 dist-test-slave-qb36 krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887796, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:56.055768 22501 init.cc:377] Logged in from keytab as oryx/127.14.153.62@KRBTEST.COM (short username oryx)
I20260619 16:49:56.057384 22501 webserver.cc:460] Webserver started at http://127.14.153.62:40531/ using document root <none> and password file <none>
I20260619 16:49:56.058153 22501 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:56.058290 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:56.058627 22501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:56.061760 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "8f34fc1d779b4f81b2f074cd8cf80a76"
format_stamp: "Formatted at 2026-06-19 16:49:56 on dist-test-slave-qb36"
server_key: "68fd0c7889bbe19bad0cb3e7d340d28d"
server_key_iv: "3adfcc8d8d3526f37d062a06e8b2c8ec"
server_key_version: "encryptionkey@0"
I20260619 16:49:56.062589 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "8f34fc1d779b4f81b2f074cd8cf80a76"
format_stamp: "Formatted at 2026-06-19 16:49:56 on dist-test-slave-qb36"
server_key: "68fd0c7889bbe19bad0cb3e7d340d28d"
server_key_iv: "3adfcc8d8d3526f37d062a06e8b2c8ec"
server_key_version: "encryptionkey@0"
I20260619 16:49:56.067879 22501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:49:56.071432 22516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:56.072810 22501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:49:56.073012 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "8f34fc1d779b4f81b2f074cd8cf80a76"
format_stamp: "Formatted at 2026-06-19 16:49:56 on dist-test-slave-qb36"
server_key: "68fd0c7889bbe19bad0cb3e7d340d28d"
server_key_iv: "3adfcc8d8d3526f37d062a06e8b2c8ec"
server_key_version: "encryptionkey@0"
I20260619 16:49:56.073256 22501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:56.087401 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:56.091765 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:56.091993 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:56.113392 22501 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:36287
I20260619 16:49:56.113417 22577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:36287 every 8 connection(s)
I20260619 16:49:56.114866 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:56.116380 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 22501
I20260619 16:49:56.116590 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:49:56.117406 14948 external_mini_cluster.cc:1428] Setting key 42d72652a391cbb1872699cdf96af8a7
I20260619 16:49:56.119860 22578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:49:56 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:56.132859 22578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76: Bootstrap starting.
I20260619 16:49:56.137187 22578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:56.138466 22578 log.cc:824] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:56.142032 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:56.119488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.14.153.62:36287)
0619 16:49:56.120264 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:56.120276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:56.120317 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:49:56.121902 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:56.121924 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:56.122335 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:56.122683 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:56.124470 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.125891 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.127646 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.128377 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.128534 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.133107 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:56.133116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:56.133135 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:56.133178 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:56.137094 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:56.137940 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:56.137948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:56.137957 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:56.138170 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:56.138555 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:56.138560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:56.138563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:56.139375 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:56.139605 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:56.140252 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 16:49:56.140684 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":364,"threads_started":1}
I20260619 16:49:56.142081 22578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76: No bootstrap required, opened a new log
I20260619 16:49:56.150286 22578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f34fc1d779b4f81b2f074cd8cf80a76" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36287 } }
I20260619 16:49:56.150630 22578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:56.150702 22578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f34fc1d779b4f81b2f074cd8cf80a76, State: Initialized, Role: FOLLOWER
I20260619 16:49:56.151270 22578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f34fc1d779b4f81b2f074cd8cf80a76" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36287 } }
I20260619 16:49:56.151465 22578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:56.151561 22578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:56.151716 22578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:56.154131 22578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f34fc1d779b4f81b2f074cd8cf80a76" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36287 } }
I20260619 16:49:56.154662 22578 leader_election.cc:302] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f34fc1d779b4f81b2f074cd8cf80a76; no voters:
I20260619 16:49:56.155380 22578 leader_election.cc:288] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:56.155552 22583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:56.156258 22583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [term 1 LEADER]: Becoming Leader. State: Replica: 8f34fc1d779b4f81b2f074cd8cf80a76, State: Running, Role: LEADER
I20260619 16:49:56.156843 22583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f34fc1d779b4f81b2f074cd8cf80a76" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36287 } }
I20260619 16:49:56.157523 22578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:56.159513 22585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f34fc1d779b4f81b2f074cd8cf80a76. Latest consensus state: current_term: 1 leader_uuid: "8f34fc1d779b4f81b2f074cd8cf80a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f34fc1d779b4f81b2f074cd8cf80a76" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36287 } } }
I20260619 16:49:56.159899 22585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:56.160609 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f34fc1d779b4f81b2f074cd8cf80a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f34fc1d779b4f81b2f074cd8cf80a76" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36287 } } }
I20260619 16:49:56.160954 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:56.162858 22589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:56.168414 22589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:56.177556 22589 catalog_manager.cc:1347] Generated new cluster ID: d5ca3017532c4b5f853aefeacc80de89
I20260619 16:49:56.177698 22589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:56.213551 22589 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:56.215546 22589 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:56.238392 22589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76: Generated new TSK 0
I20260619 16:49:56.239369 22589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab.
Entry for principal oryx/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab.
I20260619 16:49:56.302770 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:36287
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab
--principal=oryx/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:56.636621 22604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:56.636922 22604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:56.637089 22604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:56.645056 22604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:56.645170 22604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:56.645354 22604 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:56.645484 22604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:49:56.656107 22604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab
--principal=oryx/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.153.62:36287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:56.657554 22604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:56.659204 22604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:56.673098 22610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:56.673323 22612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:56.674216 22609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:56.674762 22604 server_base.cc:1034] running on GCE node
I20260619 16:49:56.675661 22604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:56.676615 22604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:56.677872 22604 hybrid_clock.cc:648] HybridClock initialized: now 1781887796677789 us; error 78 us; skew 500 ppm
I20260619 16:49:56.678211 22604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:56 dist-test-slave-qb36 krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887796, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:56.681344 22604 init.cc:377] Logged in from keytab as oryx/127.14.153.1@KRBTEST.COM (short username oryx)
I20260619 16:49:56.682713 22604 webserver.cc:460] Webserver started at http://127.14.153.1:45941/ using document root <none> and password file <none>
I20260619 16:49:56.683481 22604 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:56.683609 22604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:56.683929 22604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:56.686808 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "f92093dd6b35417f8eee869cf3bc4aeb"
format_stamp: "Formatted at 2026-06-19 16:49:56 on dist-test-slave-qb36"
server_key: "c41345538ac019548390766388814cb5"
server_key_iv: "8dbd02ccbdee1b9e301b207d3c8c5d29"
server_key_version: "encryptionkey@0"
I20260619 16:49:56.687649 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "f92093dd6b35417f8eee869cf3bc4aeb"
format_stamp: "Formatted at 2026-06-19 16:49:56 on dist-test-slave-qb36"
server_key: "c41345538ac019548390766388814cb5"
server_key_iv: "8dbd02ccbdee1b9e301b207d3c8c5d29"
server_key_version: "encryptionkey@0"
I20260619 16:49:56.692785 22604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:49:56.696098 22619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:56.697360 22604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:49:56.697532 22604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "f92093dd6b35417f8eee869cf3bc4aeb"
format_stamp: "Formatted at 2026-06-19 16:49:56 on dist-test-slave-qb36"
server_key: "c41345538ac019548390766388814cb5"
server_key_iv: "8dbd02ccbdee1b9e301b207d3c8c5d29"
server_key_version: "encryptionkey@0"
I20260619 16:49:56.697770 22604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:56.721741 22604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:56.726011 22604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:56.726297 22604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:56.729185 22604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:56.729285 22604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:56.729417 22604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:56.729497 22604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:56 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887796, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:56.753839 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:56.729789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:36481 (local address 127.14.153.62:36287)
0619 16:49:56.730409 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:56.730414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:56.731536 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 16:49:56.733499 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:56.733505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:56.733593 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:56.733715 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:56.736096 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.737423 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.740024 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.740879 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.741031 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.745865 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:56.745874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:56.745882 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:56.745936 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:56.749849 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:56.751142 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:56.751150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:56.751156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:56.751332 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:56.752049 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:56.752053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:56.752055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:56.752325 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:56.752484 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:56.753235 (+ 751us) server_negotiation.cc:299] Negotiation successful
0619 16:49:56.753448 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":415,"threads_started":1}
I20260619 16:49:56.755278 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:56.730475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36287 (local address 127.14.153.1:36481)
0619 16:49:56.731371 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:56.731410 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:56.733226 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:56.733992 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:56.734006 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:56.734836 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:56.735871 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.735908 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.737631 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.737643 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.739881 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.739893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.741198 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.741212 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.741538 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.742440 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:56.742462 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:56.745577 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:56.750082 (+ 4505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:56.750094 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:56.750122 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:56.750933 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:56.751510 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:56.751515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:56.751521 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:56.751885 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:56.752462 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:56.752471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:56.752858 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 16:49:56.753141 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 16:49:56.753685 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":344,"threads_started":1}
I20260619 16:49:56.779237 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:56.761409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:52183 (local address 127.14.153.62:36287)
0619 16:49:56.761569 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:56.761573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:56.766890 (+ 5317us) server_negotiation.cc:407] Connection header received
0619 16:49:56.767138 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:56.767144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:56.767218 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:56.767324 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:56.768258 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.768970 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.770298 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.770715 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.770858 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.773330 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:56.773335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:56.773338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:56.773368 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:56.776261 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:56.777248 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:56.777251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:56.777254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:56.777380 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:56.777755 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:56.777760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:56.777764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:56.778026 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:56.778187 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:56.778605 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 16:49:56.778823 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:49:56.779337 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:56.761484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36287 (local address 127.14.153.1:52183)
0619 16:49:56.766771 (+ 5287us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:56.766789 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:56.766965 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:56.767355 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:56.767359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:56.767831 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:56.768111 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.768120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.769115 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.769120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.770151 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.770162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.770902 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.770908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.771135 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.771862 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:56.771882 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:56.773138 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:56.776395 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:56.776404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:56.776408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:56.777077 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:56.777490 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:56.777496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:56.777499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:56.777640 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:56.778150 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:56.778155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:56.778359 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 16:49:56.778513 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 16:49:56.778832 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5153,"thread_start_us":4965,"threads_started":1}
I20260619 16:49:56.789422 22604 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:34983
I20260619 16:49:56.789451 22739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:34983 every 8 connection(s)
I20260619 16:49:56.790693 22604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:56.792744 22740 heartbeater.cc:347] Connected to a master server at 127.14.153.62:36287
I20260619 16:49:56.792966 22740 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:56.793618 22740 heartbeater.cc:511] Master 127.14.153.62:36287 requested a full tablet report, sending...
I20260619 16:49:56.795554 22533 ts_manager.cc:194] Registered new tserver with Master: f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983)
I20260619 16:49:56.796949 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 22604
I20260619 16:49:56.796957 22533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:36481
I20260619 16:49:56.797170 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:56.797883 14948 external_mini_cluster.cc:1428] Setting key ee396f79a0ea337ea9ba5c49a2ab669f
WARNING: no policy specified for oryx/127.14.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab.
Entry for principal oryx/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab.
I20260619 16:49:56.844676 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:36287
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab
--principal=oryx/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:57.171011 22744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:57.171336 22744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:57.171520 22744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:57.178969 22744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:57.179152 22744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:57.179319 22744 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:57.179450 22744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:49:57.189541 22744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab
--principal=oryx/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.153.62:36287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:57.190894 22744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:57.192404 22744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:57.205994 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:57.206568 22744 server_base.cc:1034] running on GCE node
W20260619 16:49:57.205973 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:57.205981 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:57.207590 22744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:57.208521 22744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:57.209822 22744 hybrid_clock.cc:648] HybridClock initialized: now 1781887797209762 us; error 46 us; skew 500 ppm
I20260619 16:49:57.210171 22744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:57 dist-test-slave-qb36 krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:57.213431 22744 init.cc:377] Logged in from keytab as oryx/127.14.153.2@KRBTEST.COM (short username oryx)
I20260619 16:49:57.215129 22744 webserver.cc:460] Webserver started at http://127.14.153.2:45045/ using document root <none> and password file <none>
I20260619 16:49:57.215847 22744 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:57.215977 22744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:57.216308 22744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:57.219413 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "7e96c2a1fcd04b248bfa867795679918"
format_stamp: "Formatted at 2026-06-19 16:49:57 on dist-test-slave-qb36"
server_key: "1944308b9f85da01bb9fcb6985b3f11c"
server_key_iv: "cbf6a9008d561f73555141552025fc98"
server_key_version: "encryptionkey@0"
I20260619 16:49:57.220232 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "7e96c2a1fcd04b248bfa867795679918"
format_stamp: "Formatted at 2026-06-19 16:49:57 on dist-test-slave-qb36"
server_key: "1944308b9f85da01bb9fcb6985b3f11c"
server_key_iv: "cbf6a9008d561f73555141552025fc98"
server_key_version: "encryptionkey@0"
I20260619 16:49:57.225560 22744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:57.229251 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:57.230654 22744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:57.230847 22744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "7e96c2a1fcd04b248bfa867795679918"
format_stamp: "Formatted at 2026-06-19 16:49:57 on dist-test-slave-qb36"
server_key: "1944308b9f85da01bb9fcb6985b3f11c"
server_key_iv: "cbf6a9008d561f73555141552025fc98"
server_key_version: "encryptionkey@0"
I20260619 16:49:57.231091 22744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:57.253599 22744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:57.259100 22744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:57.259394 22744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:57.262269 22744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:57.262389 22744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:57.262567 22744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:57.262645 22744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:57 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:57.290730 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.263848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36287 (local address 127.14.153.2:60063)
0619 16:49:57.264837 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:57.264925 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:57.267947 (+ 3022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:57.268862 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:57.268879 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:57.269827 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:57.270841 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.270872 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.272054 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.272062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.275314 (+ 3252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.275336 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.276537 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.276545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.276972 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.278022 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:57.278067 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:57.281220 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:57.284722 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:57.284729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:57.284744 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:57.285260 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:57.285766 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:57.285769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:57.285772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:57.285976 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:57.286630 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:57.286642 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:57.287325 (+ 683us) client_negotiation.cc:769] Sending connection context
0619 16:49:57.287755 (+ 430us) client_negotiation.cc:240] Negotiation successful
0619 16:49:57.288519 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":408,"threads_started":1}
I20260619 16:49:57.292397 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.263174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:60063 (local address 127.14.153.62:36287)
0619 16:49:57.263361 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:57.263367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:57.265085 (+ 1718us) server_negotiation.cc:407] Connection header received
0619 16:49:57.268344 (+ 3259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:57.268350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:57.268422 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:57.268517 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:57.271137 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.271880 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.275549 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.276164 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.276320 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.281461 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:57.281466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:57.281470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:57.281504 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:57.284520 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:57.285446 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:57.285453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:57.285456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:57.285629 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:57.286119 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:57.286124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:57.286128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:57.286462 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:57.286643 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:57.291573 (+ 4930us) server_negotiation.cc:299] Negotiation successful
0619 16:49:57.291902 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:49:57.315479 22744 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:44107
I20260619 16:49:57.315497 22878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:44107 every 8 connection(s)
I20260619 16:49:57.317174 22744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:57.319633 22879 heartbeater.cc:347] Connected to a master server at 127.14.153.62:36287
I20260619 16:49:57.319893 22879 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:57.320616 22879 heartbeater.cc:511] Master 127.14.153.62:36287 requested a full tablet report, sending...
I20260619 16:49:57.322359 22533 ts_manager.cc:194] Registered new tserver with Master: 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107)
I20260619 16:49:57.323606 22533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:60063
I20260619 16:49:57.326563 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 22744
I20260619 16:49:57.326623 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.310569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:43115 (local address 127.14.153.62:36287)
0619 16:49:57.310820 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:57.310830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:57.311465 (+ 635us) server_negotiation.cc:407] Connection header received
0619 16:49:57.311619 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:57.311629 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:57.311739 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:57.311869 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:57.313556 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.314744 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.316837 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.317277 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.317444 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.320197 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:57.320202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:57.320205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:57.320239 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:57.323546 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:57.324567 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:57.324573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:57.324577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:57.324791 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:57.325204 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:57.325207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:57.325209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:57.325514 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:57.325621 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:57.326055 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 16:49:57.326273 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:49:57.326740 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:57.326800 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.310521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36287 (local address 127.14.153.2:43115)
0619 16:49:57.311309 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:57.311332 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:57.311498 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:57.311907 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:57.311914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:57.312686 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:57.313337 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.313357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.314957 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.314962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.316674 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.316692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.317489 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.317495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.317748 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.318897 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:57.318924 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:57.320032 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:57.323720 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:57.323724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:57.323727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:57.324399 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:57.324919 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:57.324924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:57.324928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:57.325088 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:57.325634 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:57.325639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:57.325823 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 16:49:57.325965 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:49:57.326276 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":386,"threads_started":1}
I20260619 16:49:57.327278 14948 external_mini_cluster.cc:1428] Setting key 336e1aa1b5aff02b91b5e143af99db36
WARNING: no policy specified for oryx/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab.
Entry for principal oryx/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab.
I20260619 16:49:57.375013 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:36287
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab
--principal=oryx/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:57.706853 22883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:57.707237 22883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:57.707340 22883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:57.714509 22883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:57.714632 22883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:57.714718 22883 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:57.714802 22883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:49:57.724768 22883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab
--principal=oryx/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.153.62:36287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:49:57.726050 22883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:57.727448 22883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:57.740913 22891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:57.741000 22889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:57.740979 22888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:57.741793 22883 server_base.cc:1034] running on GCE node
I20260619 16:49:57.742399 22883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:57.743325 22883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:57.744541 22883 hybrid_clock.cc:648] HybridClock initialized: now 1781887797744479 us; error 34 us; skew 500 ppm
I20260619 16:49:57.744903 22883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:49:57 dist-test-slave-qb36 krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:57.748162 22883 init.cc:377] Logged in from keytab as oryx/127.14.153.3@KRBTEST.COM (short username oryx)
I20260619 16:49:57.749779 22883 webserver.cc:460] Webserver started at http://127.14.153.3:43199/ using document root <none> and password file <none>
I20260619 16:49:57.750474 22883 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:57.750562 22883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:57.750849 22883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:57.754110 22883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "668a10e9b0214ac2a776638d52fdab9f"
format_stamp: "Formatted at 2026-06-19 16:49:57 on dist-test-slave-qb36"
server_key: "aac72889c5e4d7ab64fd4040c22997cd"
server_key_iv: "93af8887ee2eb6b758d02025ea93f5f6"
server_key_version: "encryptionkey@0"
I20260619 16:49:57.755111 22883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "668a10e9b0214ac2a776638d52fdab9f"
format_stamp: "Formatted at 2026-06-19 16:49:57 on dist-test-slave-qb36"
server_key: "aac72889c5e4d7ab64fd4040c22997cd"
server_key_iv: "93af8887ee2eb6b758d02025ea93f5f6"
server_key_version: "encryptionkey@0"
I20260619 16:49:57.760679 22883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:49:57.764358 22898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:57.765846 22883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:57.766023 22883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "668a10e9b0214ac2a776638d52fdab9f"
format_stamp: "Formatted at 2026-06-19 16:49:57 on dist-test-slave-qb36"
server_key: "aac72889c5e4d7ab64fd4040c22997cd"
server_key_iv: "93af8887ee2eb6b758d02025ea93f5f6"
server_key_version: "encryptionkey@0"
I20260619 16:49:57.766263 22883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:57.791253 22883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:57.795943 22883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:57.796226 22883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:57.798748 22883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:57.798878 22883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:57.799100 22883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:57.799183 22883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 16:49:57.799894 22740 heartbeater.cc:503] Master 127.14.153.62:36287 was elected leader, sending a full tablet report...
Jun 19 16:49:57 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:57.821246 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.799591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:43231 (local address 127.14.153.62:36287)
0619 16:49:57.799797 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:57.799803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:57.801315 (+ 1512us) server_negotiation.cc:407] Connection header received
0619 16:49:57.803628 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:57.803634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:57.803729 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:57.803848 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:57.805938 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.806682 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.808855 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.809281 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.809412 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.814298 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:57.814305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:57.814311 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:57.814377 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:57.817355 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:57.818150 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:57.818155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:57.818159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:57.818309 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:57.818748 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:57.818753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:57.818757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:57.819128 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:57.819285 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:57.820373 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0619 16:49:57.820719 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:49:57.821872 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.800298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36287 (local address 127.14.153.3:43231)
0619 16:49:57.801163 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:57.801214 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:57.803330 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:57.804096 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:57.804104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:57.804825 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:57.805747 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.805778 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.806836 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.806841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.808703 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.808722 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.809517 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.809524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.809871 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.810693 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:57.810737 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:57.814025 (+ 3288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:57.817504 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:57.817513 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:57.817530 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:57.818021 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:57.818423 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:57.818428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:57.818432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:57.818631 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:57.819259 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:57.819268 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:57.819834 (+ 566us) client_negotiation.cc:769] Sending connection context
0619 16:49:57.820172 (+ 338us) client_negotiation.cc:240] Negotiation successful
0619 16:49:57.820693 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":527,"threads_started":1}
I20260619 16:49:57.843140 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.829329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36287 (local address 127.14.153.3:47893)
0619 16:49:57.830040 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:57.830062 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:57.830239 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:57.830729 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:57.830735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:57.831337 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:57.831887 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.831907 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.833225 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.833234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.834372 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.834384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.835277 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.835288 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.835502 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.836262 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:57.836279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:57.836974 (+ 695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:57.840248 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:57.840253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:57.840257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:57.840814 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:57.841311 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:57.841314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:57.841320 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:57.841491 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:57.842213 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:57.842220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:57.842405 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 16:49:57.842522 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:49:57.842758 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":407,"threads_started":1}
I20260619 16:49:57.843508 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.829349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:47893 (local address 127.14.153.62:36287)
0619 16:49:57.829600 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:57.829609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:57.830178 (+ 569us) server_negotiation.cc:407] Connection header received
0619 16:49:57.830401 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:57.830409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:57.830510 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:57.830634 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:57.832054 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.833084 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.834518 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.835018 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.835210 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.837257 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:57.837263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:57.837275 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:57.837304 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:57.840105 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:57.841011 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:57.841020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:57.841025 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:57.841179 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:57.841628 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:57.841635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:57.841640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:57.842062 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:57.842250 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:57.842683 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 16:49:57.842992 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:49:57.862519 22883 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:39245
I20260619 16:49:57.862597 23017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:39245 every 8 connection(s)
I20260619 16:49:57.864020 22883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:57.866247 23018 heartbeater.cc:347] Connected to a master server at 127.14.153.62:36287
I20260619 16:49:57.866477 23018 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:57.867192 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 22883
I20260619 16:49:57.867326 23018 heartbeater.cc:511] Master 127.14.153.62:36287 requested a full tablet report, sending...
I20260619 16:49:57.867363 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:57.867913 14948 external_mini_cluster.cc:1428] Setting key 80ed02a3efcefd814ed76a6ae803bde7
I20260619 16:49:57.868845 22533 ts_manager.cc:194] Registered new tserver with Master: 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3:39245)
I20260619 16:49:57.870040 22533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:43231
I20260619 16:49:57.870990 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:49:57 dist-test-slave-qb36 krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:49:57 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:57 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:58.327622 22879 heartbeater.cc:503] Master 127.14.153.62:36287 was elected leader, sending a full tablet report...
I20260619 16:49:58.872620 23018 heartbeater.cc:503] Master 127.14.153.62:36287 was elected leader, sending a full tablet report...
W20260619 16:49:58.962548 23026 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.048s user 0.007s sys 0.007s
W20260619 16:49:58.963714 22628 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:57.909031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.14.153.62:36287)
0619 16:49:57.909235 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:57.909239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:57.909571 (+ 332us) server_negotiation.cc:407] Connection header received
0619 16:49:57.909666 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:57.909670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:57.909738 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:57.909835 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:57.911168 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.911945 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.913177 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.913617 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.913749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.963405 (+1049656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41148: BlockingRecv error: recv got EOF from 127.0.0.1:41148 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 16:49:58 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:58.985329 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.968028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.14.153.62:36287)
0619 16:49:58.968223 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:58.968231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:58.968519 (+ 288us) server_negotiation.cc:407] Connection header received
0619 16:49:58.968735 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:58.968743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:58.968838 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:58.968949 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:58.970192 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.970997 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.972284 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.972698 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.972834 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.980346 (+ 7512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:58.980351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:58.980354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:58.980390 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:58.982807 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:58.983549 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:58.983554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:58.983556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:58.983667 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:58.984038 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:58.984041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:58.984043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:58.984304 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:58.984512 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:58.984826 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 16:49:58.984996 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:49:58.989717 22532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:58.996815 22532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:59.035426 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.009143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.14.153.3:39245)
0619 16:49:59.025819 (+ 16676us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.025832 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.025868 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:49:59.025984 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.025995 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.026313 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.026544 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:59.027663 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.029404 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.032375 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.033891 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.034033 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.034291 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.034903 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.035140 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16483,"thread_start_us":17589,"threads_started":1}
I20260619 16:49:59.035440 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.009028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:39245 (local address 127.0.0.1:36020)
0619 16:49:59.024853 (+ 15825us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.024910 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.025175 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.026639 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.026652 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.026708 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:59.027357 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.027377 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.029562 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.029567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.032180 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.032193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.034424 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.034434 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.034618 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.034689 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.034845 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.035037 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15452,"thread_start_us":15290,"threads_started":1}
I20260619 16:49:59.037288 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.011428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:34983 (local address 127.0.0.1:33902)
0619 16:49:59.029003 (+ 17575us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.029020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.029132 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.029996 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.030000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.030024 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:59.030567 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.030575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.031893 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.031901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.033830 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.033842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.036254 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.036261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.036420 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.036453 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.036534 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.036658 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17468,"mutex_wait_us":99,"thread_start_us":16221,"threads_started":1}
I20260619 16:49:59.037384 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.011842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.14.153.1:34983)
0619 16:49:59.027431 (+ 15589us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.027442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.029268 (+ 1826us) server_negotiation.cc:407] Connection header received
0619 16:49:59.029366 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.029376 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.029599 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.029800 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:59.030739 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.031755 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.033985 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.035319 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.035446 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.036861 (+ 1415us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.036935 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.037098 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15408,"thread_start_us":15242,"threads_started":1}
I20260619 16:49:59.038763 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.009283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:44107 (local address 127.0.0.1:40950)
0619 16:49:59.027997 (+ 18714us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.028022 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.028172 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.029515 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.029520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.029542 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:59.030178 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.030191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.032357 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.032362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.034581 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.034596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.037162 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.037167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.037366 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.037414 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.037501 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.037653 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18537,"thread_start_us":19480,"threads_started":1}
I20260619 16:49:59.039602 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.009572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.14.153.2:44107)
0619 16:49:59.028689 (+ 19117us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.028699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.028730 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:49:59.028845 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.028854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.029111 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.029325 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:59.030389 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.032198 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.034772 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.036677 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.036818 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.037025 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.039145 (+ 2120us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.039302 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18966,"thread_start_us":18832,"threads_started":1}
I20260619 16:49:59.039351 22953 tablet_service.cc:1467] Processing CreateTablet for tablet bddae5688f7c4b8cb2adef69a51397db (DEFAULT_TABLE table=test-table [id=11554dbaef7c48b7b06020c1d8d63639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:59.040880 22953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bddae5688f7c4b8cb2adef69a51397db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.040678 22670 tablet_service.cc:1467] Processing CreateTablet for tablet bddae5688f7c4b8cb2adef69a51397db (DEFAULT_TABLE table=test-table [id=11554dbaef7c48b7b06020c1d8d63639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:59.042058 22670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bddae5688f7c4b8cb2adef69a51397db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.043058 22809 tablet_service.cc:1467] Processing CreateTablet for tablet bddae5688f7c4b8cb2adef69a51397db (DEFAULT_TABLE table=test-table [id=11554dbaef7c48b7b06020c1d8d63639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:59.044374 22809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bddae5688f7c4b8cb2adef69a51397db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.053896 23043 tablet_bootstrap.cc:492] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: Bootstrap starting.
I20260619 16:49:59.056581 23044 tablet_bootstrap.cc:492] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: Bootstrap starting.
I20260619 16:49:59.057646 23043 tablet_bootstrap.cc:654] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.059209 23043 log.cc:824] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:59.060702 23044 tablet_bootstrap.cc:654] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.062250 23044 log.cc:824] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:59.063957 23045 tablet_bootstrap.cc:492] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: Bootstrap starting.
I20260619 16:49:59.065744 23044 tablet_bootstrap.cc:492] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: No bootstrap required, opened a new log
I20260619 16:49:59.066013 23044 ts_tablet_manager.cc:1397] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260619 16:49:59.068663 23045 tablet_bootstrap.cc:654] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.070698 23045 log.cc:824] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:59.071904 23043 tablet_bootstrap.cc:492] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: No bootstrap required, opened a new log
I20260619 16:49:59.072209 23043 ts_tablet_manager.cc:1397] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20260619 16:49:59.073545 23045 tablet_bootstrap.cc:492] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: No bootstrap required, opened a new log
I20260619 16:49:59.073810 23045 ts_tablet_manager.cc:1397] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 16:49:59.075730 23044 raft_consensus.cc:348] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } }
I20260619 16:49:59.076185 23044 raft_consensus.cc:374] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.076345 23044 raft_consensus.cc:729] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92093dd6b35417f8eee869cf3bc4aeb, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.077076 23044 consensus_queue.cc:260] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } }
I20260619 16:49:59.078981 23044 ts_tablet_manager.cc:1428] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260619 16:49:59.082122 23045 raft_consensus.cc:348] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } }
I20260619 16:49:59.082580 23045 raft_consensus.cc:374] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.082724 23045 raft_consensus.cc:729] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e96c2a1fcd04b248bfa867795679918, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.082917 23043 raft_consensus.cc:348] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } }
I20260619 16:49:59.083416 23043 raft_consensus.cc:374] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.083521 23043 raft_consensus.cc:729] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668a10e9b0214ac2a776638d52fdab9f, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.083405 23045 consensus_queue.cc:260] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } }
I20260619 16:49:59.084095 23043 consensus_queue.cc:260] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } }
I20260619 16:49:59.085815 23043 ts_tablet_manager.cc:1428] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260619 16:49:59.086493 23045 ts_tablet_manager.cc:1428] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260619 16:49:59.116379 23019 tablet.cc:2369] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:59.164505 23051 raft_consensus.cc:482] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:59.164826 23051 raft_consensus.cc:504] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } }
I20260619 16:49:59.166533 23051 leader_election.cc:288] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107), f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983)
I20260619 16:49:59.173278 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.166961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:54419 (local address 127.14.153.2:44107)
0619 16:49:59.167221 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.167228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.168054 (+ 826us) server_negotiation.cc:407] Connection header received
0619 16:49:59.168269 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.168274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.168340 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.168434 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:59.168985 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.169753 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.171693 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.172415 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.172593 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.172668 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.172922 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.173042 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 16:49:59.173436 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.167088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:44107 (local address 127.14.153.3:54419)
0619 16:49:59.167938 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.167955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.168124 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.168459 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.168463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.168487 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:59.168844 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.168856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.169888 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.169896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.171526 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.171536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.172607 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.172613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.172769 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.172792 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.172897 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.173017 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":1,"thread_start_us":578,"threads_started":1}
I20260619 16:49:59.173959 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.167004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:46729 (local address 127.14.153.1:34983)
0619 16:49:59.167241 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.167247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.168117 (+ 870us) server_negotiation.cc:407] Connection header received
0619 16:49:59.168331 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.168341 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.168413 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.168521 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:59.169179 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.170079 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.172103 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.172921 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.173123 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.173204 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.173597 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.173733 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 16:49:59.174008 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.167119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:34983 (local address 127.14.153.3:46729)
0619 16:49:59.167995 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.168012 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.168184 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.168541 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.168545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.168588 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:59.169016 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.169024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.170215 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.170223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.171938 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.171949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.173175 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.173183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.173360 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.173416 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.173550 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.173678 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":445,"threads_started":1}
I20260619 16:49:59.174633 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddae5688f7c4b8cb2adef69a51397db" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e96c2a1fcd04b248bfa867795679918" is_pre_election: true
I20260619 16:49:59.175086 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddae5688f7c4b8cb2adef69a51397db" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" is_pre_election: true
I20260619 16:49:59.175261 22832 raft_consensus.cc:2454] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 0.
I20260619 16:49:59.175601 22690 raft_consensus.cc:2454] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 0.
I20260619 16:49:59.175953 22902 leader_election.cc:302] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, 7e96c2a1fcd04b248bfa867795679918; no voters:
I20260619 16:49:59.176414 23051 raft_consensus.cc:2781] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:59.176553 23051 raft_consensus.cc:482] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:59.176666 23051 raft_consensus.cc:3037] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.179006 23051 raft_consensus.cc:504] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } }
I20260619 16:49:59.179674 23051 leader_election.cc:288] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 election: Requested vote from peers 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107), f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983)
I20260619 16:49:59.180182 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddae5688f7c4b8cb2adef69a51397db" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb"
I20260619 16:49:59.180183 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bddae5688f7c4b8cb2adef69a51397db" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e96c2a1fcd04b248bfa867795679918"
I20260619 16:49:59.180392 22690 raft_consensus.cc:3037] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.180394 22832 raft_consensus.cc:3037] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.183576 22690 raft_consensus.cc:2454] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 1.
I20260619 16:49:59.183576 22832 raft_consensus.cc:2454] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 1.
I20260619 16:49:59.184002 22899 leader_election.cc:302] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, f92093dd6b35417f8eee869cf3bc4aeb; no voters:
I20260619 16:49:59.184310 23051 raft_consensus.cc:2781] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:59.184859 23051 raft_consensus.cc:686] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 1 LEADER]: Becoming Leader. State: Replica: 668a10e9b0214ac2a776638d52fdab9f, State: Running, Role: LEADER
I20260619 16:49:59.185431 23051 consensus_queue.cc:237] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } }
I20260619 16:49:59.190012 22533 catalog_manager.cc:5495] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3). New cstate: current_term: 1 leader_uuid: "668a10e9b0214ac2a776638d52fdab9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:59.234983 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.229619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.14.153.3:39245)
0619 16:49:59.229834 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.229841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.229888 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:49:59.230222 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.230227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.230313 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.230477 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:59.230842 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.231540 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.232865 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.233255 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.233387 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.233799 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:59.234360 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:59.234544 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.234602 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.234680 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:49:59.242789 22690 raft_consensus.cc:1261] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Refusing update from remote peer 668a10e9b0214ac2a776638d52fdab9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:59.242794 22832 raft_consensus.cc:1261] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Refusing update from remote peer 668a10e9b0214ac2a776638d52fdab9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:59.243819 23054 consensus_queue.cc:1035] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:59.244179 23051 consensus_queue.cc:1035] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:59.254699 23057 mvcc.cc:205] Tried to move back new op lower bound from 7298612425689395200 to 7298612425467789312. Current Snapshot: MvccSnapshot[applied={T|T < 7298612425689395200}]
I20260619 16:49:59.258049 23058 mvcc.cc:205] Tried to move back new op lower bound from 7298612425689395200 to 7298612425467789312. Current Snapshot: MvccSnapshot[applied={T|T < 7298612425689395200}]
I20260619 16:49:59.258131 23059 mvcc.cc:205] Tried to move back new op lower bound from 7298612425689395200 to 7298612425467789312. Current Snapshot: MvccSnapshot[applied={T|T < 7298612425689395200}]
I20260619 16:49:59.270231 22533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:59.270493 22533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:59.273896 22533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76: Sending DeleteTablet for 3 replicas of tablet bddae5688f7c4b8cb2adef69a51397db
I20260619 16:49:59.274930 22953 tablet_service.cc:1514] Processing DeleteTablet for tablet bddae5688f7c4b8cb2adef69a51397db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:59 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:36020
I20260619 16:49:59.275056 22809 tablet_service.cc:1514] Processing DeleteTablet for tablet bddae5688f7c4b8cb2adef69a51397db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:59 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260619 16:49:59.276778 22670 tablet_service.cc:1514] Processing DeleteTablet for tablet bddae5688f7c4b8cb2adef69a51397db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:59 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:33902
I20260619 16:49:59.276823 22533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:59.277738 22533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:59.281280 23067 tablet_replica.cc:330] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: stopping tablet replica
I20260619 16:49:59.281817 23067 raft_consensus.cc:2229] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:59.282544 23067 raft_consensus.cc:2258] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:59.284787 23067 ts_tablet_manager.cc:1905] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:59.287413 22669 tablet_service.cc:1467] Processing CreateTablet for tablet c257527ad0c7468cb983420a33d87c95 (DEFAULT_TABLE table=test-table [id=50276153793d4bd8abc375c8517b95d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:59.287741 22951 tablet_service.cc:1467] Processing CreateTablet for tablet c257527ad0c7468cb983420a33d87c95 (DEFAULT_TABLE table=test-table [id=50276153793d4bd8abc375c8517b95d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:59.288152 22669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c257527ad0c7468cb983420a33d87c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.288182 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c257527ad0c7468cb983420a33d87c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.290521 23069 tablet_replica.cc:330] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: stopping tablet replica
I20260619 16:49:59.291177 23069 raft_consensus.cc:2229] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:59.291682 23069 raft_consensus.cc:2258] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:59.293651 23043 tablet_bootstrap.cc:492] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: Bootstrap starting.
I20260619 16:49:59.295377 23043 tablet_bootstrap.cc:654] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.295387 23069 ts_tablet_manager.cc:1905] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:59.297461 22808 tablet_service.cc:1467] Processing CreateTablet for tablet c257527ad0c7468cb983420a33d87c95 (DEFAULT_TABLE table=test-table [id=50276153793d4bd8abc375c8517b95d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:59.298055 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c257527ad0c7468cb983420a33d87c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.299850 23068 tablet_replica.cc:330] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: stopping tablet replica
I20260619 16:49:59.300289 23068 raft_consensus.cc:2229] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:59.300562 23067 ts_tablet_manager.cc:1918] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:59.300733 23067 log.cc:1196] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/bddae5688f7c4b8cb2adef69a51397db
I20260619 16:49:59.300946 23068 raft_consensus.cc:2258] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:59.301275 23067 ts_tablet_manager.cc:1939] T bddae5688f7c4b8cb2adef69a51397db P 668a10e9b0214ac2a776638d52fdab9f: Deleting consensus metadata
I20260619 16:49:59.301415 23044 tablet_bootstrap.cc:492] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: Bootstrap starting.
I20260619 16:49:59.302984 22518 catalog_manager.cc:4841] TS 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3:39245): tablet bddae5688f7c4b8cb2adef69a51397db (table test-table [id=11554dbaef7c48b7b06020c1d8d63639]) successfully deleted
I20260619 16:49:59.303355 23068 ts_tablet_manager.cc:1905] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:59.303448 23044 tablet_bootstrap.cc:654] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.306138 23043 tablet_bootstrap.cc:492] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: No bootstrap required, opened a new log
I20260619 16:49:59.306285 23043 ts_tablet_manager.cc:1397] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260619 16:49:59.307106 23043 raft_consensus.cc:348] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.307431 23043 raft_consensus.cc:374] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.307575 23043 raft_consensus.cc:729] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668a10e9b0214ac2a776638d52fdab9f, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.307694 23069 ts_tablet_manager.cc:1918] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:59.307911 23069 log.cc:1196] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/bddae5688f7c4b8cb2adef69a51397db
I20260619 16:49:59.307924 23043 consensus_queue.cc:260] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.309190 23043 ts_tablet_manager.cc:1428] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:59.309626 23069 ts_tablet_manager.cc:1939] T bddae5688f7c4b8cb2adef69a51397db P f92093dd6b35417f8eee869cf3bc4aeb: Deleting consensus metadata
I20260619 16:49:59.310818 23045 tablet_bootstrap.cc:492] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: Bootstrap starting.
I20260619 16:49:59.311506 22517 catalog_manager.cc:4841] TS f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983): tablet bddae5688f7c4b8cb2adef69a51397db (table test-table [id=11554dbaef7c48b7b06020c1d8d63639]) successfully deleted
I20260619 16:49:59.312467 23045 tablet_bootstrap.cc:654] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.313017 23068 ts_tablet_manager.cc:1918] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:59.313177 23068 log.cc:1196] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/bddae5688f7c4b8cb2adef69a51397db
I20260619 16:49:59.313824 23068 ts_tablet_manager.cc:1939] T bddae5688f7c4b8cb2adef69a51397db P 7e96c2a1fcd04b248bfa867795679918: Deleting consensus metadata
I20260619 16:49:59.314538 23045 tablet_bootstrap.cc:492] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: No bootstrap required, opened a new log
I20260619 16:49:59.314690 23045 ts_tablet_manager.cc:1397] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:49:59.315584 23045 raft_consensus.cc:348] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.315856 23045 raft_consensus.cc:374] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.315976 23045 raft_consensus.cc:729] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e96c2a1fcd04b248bfa867795679918, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.316316 23045 consensus_queue.cc:260] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.317340 22520 catalog_manager.cc:4841] TS 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107): tablet bddae5688f7c4b8cb2adef69a51397db (table test-table [id=11554dbaef7c48b7b06020c1d8d63639]) successfully deleted
I20260619 16:49:59.318959 23045 ts_tablet_manager.cc:1428] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20260619 16:49:59.320364 22880 tablet.cc:2369] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:59.322551 23044 tablet_bootstrap.cc:492] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: No bootstrap required, opened a new log
I20260619 16:49:59.322719 23044 ts_tablet_manager.cc:1397] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20260619 16:49:59.323581 23044 raft_consensus.cc:348] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.323824 23044 raft_consensus.cc:374] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.323920 23044 raft_consensus.cc:729] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92093dd6b35417f8eee869cf3bc4aeb, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.324183 23044 consensus_queue.cc:260] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.324862 23044 ts_tablet_manager.cc:1428] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260619 16:49:59.544503 22741 tablet.cc:2369] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:59.631019 23050 raft_consensus.cc:482] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:59.631333 23050 raft_consensus.cc:504] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.632925 23050 leader_election.cc:288] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3:39245)
I20260619 16:49:59.639885 23051 raft_consensus.cc:482] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:59.640101 23051 raft_consensus.cc:504] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.640834 23051 leader_election.cc:288] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107)
I20260619 16:49:59.641219 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.633909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:36187 (local address 127.14.153.1:34983)
0619 16:49:59.634098 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.634116 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.634919 (+ 803us) server_negotiation.cc:407] Connection header received
0619 16:49:59.635105 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.635110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.635183 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.635291 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:59.636097 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.637150 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.639092 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.640047 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.640217 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.640288 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.640789 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.640926 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:49:59.641933 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.633340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:34983 (local address 127.14.153.2:36187)
0619 16:49:59.634803 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.634821 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.634958 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.635333 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.635343 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.635372 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:59.635891 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.635904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.637278 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.637283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.638915 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.638926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.640425 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.640431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.640601 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.640636 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.641394 (+ 758us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.641539 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"mutex_wait_us":63,"thread_start_us":766,"threads_started":1}
I20260619 16:49:59.641777 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c257527ad0c7468cb983420a33d87c95" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" is_pre_election: true
I20260619 16:49:59.642076 22689 raft_consensus.cc:2454] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 0.
I20260619 16:49:59.642215 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.633426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:44013 (local address 127.14.153.3:39245)
0619 16:49:59.633645 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.633654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.634331 (+ 677us) server_negotiation.cc:407] Connection header received
0619 16:49:59.634685 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.634692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.634772 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.634883 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:59.635669 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.636990 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.639718 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.641001 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.641141 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.641227 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.641770 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.641907 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:49:59.642927 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c257527ad0c7468cb983420a33d87c95" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e96c2a1fcd04b248bfa867795679918" is_pre_election: true
I20260619 16:49:59.643005 22899 leader_election.cc:302] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, f92093dd6b35417f8eee869cf3bc4aeb; no voters:
I20260619 16:49:59.643200 22832 raft_consensus.cc:2454] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 0.
I20260619 16:49:59.643396 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c257527ad0c7468cb983420a33d87c95" candidate_uuid: "7e96c2a1fcd04b248bfa867795679918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" is_pre_election: true
I20260619 16:49:59.643508 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.633270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:39245 (local address 127.14.153.2:44013)
0619 16:49:59.634182 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.634207 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.634483 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.635135 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.635139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.635171 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:59.635514 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.635523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.637153 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.637160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.639532 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.639551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.641386 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.641403 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.641588 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.641620 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.642362 (+ 742us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.642506 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":65,"thread_start_us":1205,"threads_started":1}
I20260619 16:49:59.643618 22689 raft_consensus.cc:2454] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e96c2a1fcd04b248bfa867795679918 in term 0.
I20260619 16:49:59.644043 23051 raft_consensus.cc:2781] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:59.644176 23051 raft_consensus.cc:482] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:59.644095 22760 leader_election.cc:302] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e96c2a1fcd04b248bfa867795679918, f92093dd6b35417f8eee869cf3bc4aeb; no voters:
I20260619 16:49:59.644269 23051 raft_consensus.cc:3037] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.644479 23050 raft_consensus.cc:2781] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:59.644579 23050 raft_consensus.cc:482] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:59.644662 23050 raft_consensus.cc:3037] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.644639 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c257527ad0c7468cb983420a33d87c95" candidate_uuid: "7e96c2a1fcd04b248bfa867795679918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668a10e9b0214ac2a776638d52fdab9f" is_pre_election: true
I20260619 16:49:59.646610 23051 raft_consensus.cc:504] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.647115 22973 raft_consensus.cc:2379] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e96c2a1fcd04b248bfa867795679918 in current term 1: Already voted for candidate 668a10e9b0214ac2a776638d52fdab9f in this term.
I20260619 16:49:59.647716 23050 raft_consensus.cc:504] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.647807 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c257527ad0c7468cb983420a33d87c95" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb"
I20260619 16:49:59.647991 22689 raft_consensus.cc:3037] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.648753 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c257527ad0c7468cb983420a33d87c95" candidate_uuid: "7e96c2a1fcd04b248bfa867795679918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb"
I20260619 16:49:59.649401 23050 leader_election.cc:288] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [CANDIDATE]: Term 1 election: Requested vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3:39245)
I20260619 16:49:59.649770 23051 leader_election.cc:288] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 election: Requested vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107)
I20260619 16:49:59.649847 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c257527ad0c7468cb983420a33d87c95" candidate_uuid: "7e96c2a1fcd04b248bfa867795679918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668a10e9b0214ac2a776638d52fdab9f"
I20260619 16:49:59.650101 22973 raft_consensus.cc:2379] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e96c2a1fcd04b248bfa867795679918 in current term 1: Already voted for candidate 668a10e9b0214ac2a776638d52fdab9f in this term.
I20260619 16:49:59.650604 22761 leader_election.cc:302] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e96c2a1fcd04b248bfa867795679918; no voters: 668a10e9b0214ac2a776638d52fdab9f, f92093dd6b35417f8eee869cf3bc4aeb
I20260619 16:49:59.651099 22689 raft_consensus.cc:2454] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 1.
I20260619 16:49:59.651103 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c257527ad0c7468cb983420a33d87c95" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e96c2a1fcd04b248bfa867795679918"
I20260619 16:49:59.651363 22832 raft_consensus.cc:2379] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 668a10e9b0214ac2a776638d52fdab9f in current term 1: Already voted for candidate 7e96c2a1fcd04b248bfa867795679918 in this term.
I20260619 16:49:59.651526 22899 leader_election.cc:302] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, f92093dd6b35417f8eee869cf3bc4aeb; no voters:
I20260619 16:49:59.651695 23050 raft_consensus.cc:2726] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 16:49:59.651914 23051 raft_consensus.cc:2781] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:59.652115 23051 raft_consensus.cc:686] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 1 LEADER]: Becoming Leader. State: Replica: 668a10e9b0214ac2a776638d52fdab9f, State: Running, Role: LEADER
I20260619 16:49:59.652417 23051 consensus_queue.cc:237] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.655256 22533 catalog_manager.cc:5495] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3). New cstate: current_term: 1 leader_uuid: "668a10e9b0214ac2a776638d52fdab9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } health_report { overall_health: HEALTHY } } }
Jun 19 16:49:59 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887796, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.62@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:49:59.789762 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.773364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.14.153.62:36287)
0619 16:49:59.774191 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.774195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.774217 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:49:59.774372 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.774376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.774431 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.774531 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.775594 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.776366 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.777729 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.778214 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.778312 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.782806 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:59.782812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:59.782815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:59.782842 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:59.786320 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:59.787291 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:59.787297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:59.787300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:59.787449 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:59.787883 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:59.787888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:59.787891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:59.788284 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:59.788427 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.789040 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.789339 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":723,"thread_start_us":464,"threads_started":1}
I20260619 16:49:59.789811 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.773352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36287 (local address 127.0.0.1:41174)
0619 16:49:59.774131 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.774148 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.774267 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.774591 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.774595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.775276 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:59.775461 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.775469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.776515 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.776519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.777512 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.777529 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.778363 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.778368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.778503 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.779282 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:59.779300 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:59.782561 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:59.786473 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:59.786486 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:59.786495 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:59.787142 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:59.787569 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:59.787574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:59.787577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:59.787759 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:59.788491 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:59.788499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:59.788689 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.788876 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.789199 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":463,"threads_started":1}
I20260619 16:49:59.795248 22533 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:41174:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:49:59.796312 22533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:59.803256 22669 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6458966ec246caa51f0cc61a8dfc75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fb4ae4c81b948dfbd8beacd28d97da8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:59.803596 22808 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6458966ec246caa51f0cc61a8dfc75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fb4ae4c81b948dfbd8beacd28d97da8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:59.803818 22669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6458966ec246caa51f0cc61a8dfc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.804100 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6458966ec246caa51f0cc61a8dfc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.810822 22951 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6458966ec246caa51f0cc61a8dfc75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fb4ae4c81b948dfbd8beacd28d97da8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:59.811214 23045 tablet_bootstrap.cc:492] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: Bootstrap starting.
I20260619 16:49:59.811403 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6458966ec246caa51f0cc61a8dfc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.811883 23044 tablet_bootstrap.cc:492] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: Bootstrap starting.
I20260619 16:49:59.812927 23045 tablet_bootstrap.cc:654] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.813454 23044 tablet_bootstrap.cc:654] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.816308 23044 tablet_bootstrap.cc:492] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: No bootstrap required, opened a new log
I20260619 16:49:59.816480 23044 ts_tablet_manager.cc:1397] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260619 16:49:59.816768 23045 tablet_bootstrap.cc:492] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: No bootstrap required, opened a new log
I20260619 16:49:59.816927 23045 ts_tablet_manager.cc:1397] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260619 16:49:59.817200 23044 raft_consensus.cc:348] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.817442 23044 raft_consensus.cc:374] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.817524 23044 raft_consensus.cc:729] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92093dd6b35417f8eee869cf3bc4aeb, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.817626 23045 raft_consensus.cc:348] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.817858 23045 raft_consensus.cc:374] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.817929 23045 raft_consensus.cc:729] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e96c2a1fcd04b248bfa867795679918, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.817785 23044 consensus_queue.cc:260] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.818243 23045 consensus_queue.cc:260] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.818418 23044 ts_tablet_manager.cc:1428] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 16:49:59.818351 23049 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.818745 23049 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:59.818747 23050 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.819005 23050 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:59.819444 23045 ts_tablet_manager.cc:1428] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:59.823472 23085 tablet_bootstrap.cc:492] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: Bootstrap starting.
I20260619 16:49:59.824949 23085 tablet_bootstrap.cc:654] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.827137 23085 tablet_bootstrap.cc:492] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: No bootstrap required, opened a new log
I20260619 16:49:59.827257 23085 ts_tablet_manager.cc:1397] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260619 16:49:59.827806 23085 raft_consensus.cc:348] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.827977 23085 raft_consensus.cc:374] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.828035 23085 raft_consensus.cc:729] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668a10e9b0214ac2a776638d52fdab9f, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.828203 23085 consensus_queue.cc:260] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.828424 23050 raft_consensus.cc:482] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:59.828589 23050 raft_consensus.cc:504] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.828768 23085 ts_tablet_manager.cc:1428] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:59.828609 23051 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.828949 23051 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:59.829252 23050 leader_election.cc:288] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3:39245)
I20260619 16:49:59.829761 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6458966ec246caa51f0cc61a8dfc75" candidate_uuid: "7e96c2a1fcd04b248bfa867795679918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668a10e9b0214ac2a776638d52fdab9f" is_pre_election: true
I20260619 16:49:59.830013 22973 raft_consensus.cc:2454] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e96c2a1fcd04b248bfa867795679918 in term 0.
I20260619 16:49:59.829952 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6458966ec246caa51f0cc61a8dfc75" candidate_uuid: "7e96c2a1fcd04b248bfa867795679918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" is_pre_election: true
I20260619 16:49:59.830221 22689 raft_consensus.cc:2454] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e96c2a1fcd04b248bfa867795679918 in term 0.
I20260619 16:49:59.830437 22761 leader_election.cc:302] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, 7e96c2a1fcd04b248bfa867795679918; no voters:
I20260619 16:49:59.830710 23050 raft_consensus.cc:2781] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:59.830812 23050 raft_consensus.cc:482] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:59.830916 23050 raft_consensus.cc:3037] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.833385 23050 raft_consensus.cc:504] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.833962 23050 leader_election.cc:288] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [CANDIDATE]: Term 1 election: Requested vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3:39245)
I20260619 16:49:59.834379 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6458966ec246caa51f0cc61a8dfc75" candidate_uuid: "7e96c2a1fcd04b248bfa867795679918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668a10e9b0214ac2a776638d52fdab9f"
I20260619 16:49:59.834367 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6458966ec246caa51f0cc61a8dfc75" candidate_uuid: "7e96c2a1fcd04b248bfa867795679918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb"
I20260619 16:49:59.834561 22973 raft_consensus.cc:3037] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.834630 22689 raft_consensus.cc:3037] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.836663 22973 raft_consensus.cc:2454] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e96c2a1fcd04b248bfa867795679918 in term 1.
I20260619 16:49:59.836824 22689 raft_consensus.cc:2454] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e96c2a1fcd04b248bfa867795679918 in term 1.
I20260619 16:49:59.837129 22761 leader_election.cc:302] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, 7e96c2a1fcd04b248bfa867795679918; no voters:
I20260619 16:49:59.837374 23050 raft_consensus.cc:2781] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:59.837829 23050 raft_consensus.cc:686] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [term 1 LEADER]: Becoming Leader. State: Replica: 7e96c2a1fcd04b248bfa867795679918, State: Running, Role: LEADER
I20260619 16:49:59.838192 23050 consensus_queue.cc:237] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:49:59.840070 23087 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: TxnStatusTablet state changed. Reason: New leader 7e96c2a1fcd04b248bfa867795679918. Latest consensus state: current_term: 1 leader_uuid: "7e96c2a1fcd04b248bfa867795679918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.840381 23087 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:59.841135 23089 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:49:59.841337 23089 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:49:59.841465 22531 catalog_manager.cc:5495] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2). New cstate: current_term: 1 leader_uuid: "7e96c2a1fcd04b248bfa867795679918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:59.859234 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.852248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:44107 (local address 127.0.0.1:40962)
0619 16:49:59.852534 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.852563 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.852758 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.853823 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.853828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.853842 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:59.854061 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.854071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.855304 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.855311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.856428 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.856439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.857175 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.857180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.857382 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.857488 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:59.858601 (+ 1113us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:59.858622 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.858708 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.858876 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 16:49:59.859232 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.852258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.14.153.2:44107)
0619 16:49:59.853411 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.853416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.853435 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:49:59.853508 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.853511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.853587 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.853803 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:59.854239 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.855172 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.856560 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.857012 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.857163 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.857717 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:59.858452 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:59.858641 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.858733 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.858816 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":871,"threads_started":1}
I20260619 16:49:59.864639 22689 raft_consensus.cc:1261] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Refusing update from remote peer 7e96c2a1fcd04b248bfa867795679918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:59.865093 22973 raft_consensus.cc:1261] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Refusing update from remote peer 7e96c2a1fcd04b248bfa867795679918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:59.865376 23087 consensus_queue.cc:1035] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:59.866036 23087 consensus_queue.cc:1035] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:59.895586 23049 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: TxnStatusTablet state changed. Reason: New leader 7e96c2a1fcd04b248bfa867795679918. Latest consensus state: current_term: 1 leader_uuid: "7e96c2a1fcd04b248bfa867795679918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.895891 23049 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:59.899844 23094 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e96c2a1fcd04b248bfa867795679918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.900250 23094 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:59.902359 23051 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: TxnStatusTablet state changed. Reason: New leader 7e96c2a1fcd04b248bfa867795679918. Latest consensus state: current_term: 1 leader_uuid: "7e96c2a1fcd04b248bfa867795679918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.902649 23051 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:59.903513 23087 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e96c2a1fcd04b248bfa867795679918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.903748 23087 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P 7e96c2a1fcd04b248bfa867795679918: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:59.911003 23049 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e96c2a1fcd04b248bfa867795679918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.911288 23049 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P f92093dd6b35417f8eee869cf3bc4aeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:59.913214 23051 tablet_replica.cc:439] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e96c2a1fcd04b248bfa867795679918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } } }
I20260619 16:49:59.913460 23051 tablet_replica.cc:442] T 1c6458966ec246caa51f0cc61a8dfc75 P 668a10e9b0214ac2a776638d52fdab9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:59.929673 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.923639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:58427 (local address 127.14.153.2:44107)
0619 16:49:59.923858 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.923867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.924381 (+ 514us) server_negotiation.cc:407] Connection header received
0619 16:49:59.924595 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.924599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.924666 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.924739 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:59.925305 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.926071 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.927986 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.928436 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.928532 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.929027 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:59.929131 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:59.929194 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.929337 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.929396 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:49:59.929844 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.923612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:44107 (local address 127.14.153.3:58427)
0619 16:49:59.924274 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.924297 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.924450 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.924805 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.924811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.924824 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:59.925132 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.925144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.926198 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.926204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.927838 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.927854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.928583 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.928589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.928777 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.928855 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:59.929221 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:59.929232 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.929314 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.929455 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":313,"threads_started":1}
I20260619 16:49:59.939230 22832 raft_consensus.cc:1261] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Refusing update from remote peer 668a10e9b0214ac2a776638d52fdab9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:59.939299 22689 raft_consensus.cc:1261] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Refusing update from remote peer 668a10e9b0214ac2a776638d52fdab9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:59.939812 23051 consensus_queue.cc:1035] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:59.940135 23063 consensus_queue.cc:1035] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:59.979471 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.972369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:43323 (local address 127.14.153.3:39245)
0619 16:49:59.972575 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.972581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.973168 (+ 587us) server_negotiation.cc:407] Connection header received
0619 16:49:59.973378 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.973383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.973461 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.973546 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:59.974198 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.974958 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.977354 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.978034 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.978140 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.978866 (+ 726us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:59.978965 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:59.979080 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.979207 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.979266 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:49:59.979780 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.972293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:39245 (local address 127.14.153.2:43323)
0619 16:49:59.973065 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.973083 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.973236 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.973587 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.973592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.973605 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:59.974049 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.974060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.975112 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.975118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.977179 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.977195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.978407 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.978413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.978610 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.978692 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:59.979089 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:59.979108 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.979197 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.979348 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":430,"threads_started":1}
I20260619 16:50:00.011718 22531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:00.011967 22531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:00.016553 22531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76: Sending DeleteTablet for 3 replicas of tablet c257527ad0c7468cb983420a33d87c95
I20260619 16:50:00.017406 22669 tablet_service.cc:1514] Processing DeleteTablet for tablet c257527ad0c7468cb983420a33d87c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:00 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:33902
I20260619 16:50:00.017496 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet c257527ad0c7468cb983420a33d87c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:00 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:36020
I20260619 16:50:00.018613 22808 tablet_service.cc:1514] Processing DeleteTablet for tablet c257527ad0c7468cb983420a33d87c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:00 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260619 16:50:00.019913 23114 tablet_replica.cc:330] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: stopping tablet replica
I20260619 16:50:00.020198 23114 raft_consensus.cc:2229] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:00.020754 23114 raft_consensus.cc:2258] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:00.024282 23114 ts_tablet_manager.cc:1905] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:00.028800 23112 tablet_replica.cc:330] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: stopping tablet replica
I20260619 16:50:00.029095 23112 raft_consensus.cc:2229] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:00.030380 23112 raft_consensus.cc:2258] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:00.032076 23112 ts_tablet_manager.cc:1905] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:00.034530 23114 ts_tablet_manager.cc:1918] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:50:00.034690 23114 log.cc:1196] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/c257527ad0c7468cb983420a33d87c95
I20260619 16:50:00.035362 23114 ts_tablet_manager.cc:1939] T c257527ad0c7468cb983420a33d87c95 P 7e96c2a1fcd04b248bfa867795679918: Deleting consensus metadata
I20260619 16:50:00.035655 23113 tablet_replica.cc:330] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: stopping tablet replica
I20260619 16:50:00.036660 22520 catalog_manager.cc:4841] TS 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107): tablet c257527ad0c7468cb983420a33d87c95 (table test-table [id=50276153793d4bd8abc375c8517b95d1]) successfully deleted
I20260619 16:50:00.040035 23113 raft_consensus.cc:2229] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:00.040839 23113 raft_consensus.cc:2258] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:00.042093 23113 ts_tablet_manager.cc:1905] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:00.043603 23112 ts_tablet_manager.cc:1918] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:50:00.043779 23112 log.cc:1196] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/c257527ad0c7468cb983420a33d87c95
I20260619 16:50:00.044255 23112 ts_tablet_manager.cc:1939] T c257527ad0c7468cb983420a33d87c95 P 668a10e9b0214ac2a776638d52fdab9f: Deleting consensus metadata
I20260619 16:50:00.045478 22518 catalog_manager.cc:4841] TS 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3:39245): tablet c257527ad0c7468cb983420a33d87c95 (table test-table [id=50276153793d4bd8abc375c8517b95d1]) successfully deleted
I20260619 16:50:00.051067 23113 ts_tablet_manager.cc:1918] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:50:00.051250 23113 log.cc:1196] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/c257527ad0c7468cb983420a33d87c95
I20260619 16:50:00.051792 23113 ts_tablet_manager.cc:1939] T c257527ad0c7468cb983420a33d87c95 P f92093dd6b35417f8eee869cf3bc4aeb: Deleting consensus metadata
I20260619 16:50:00.053042 22517 catalog_manager.cc:4841] TS f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983): tablet c257527ad0c7468cb983420a33d87c95 (table test-table [id=50276153793d4bd8abc375c8517b95d1]) successfully deleted
Jun 19 16:50:00 dist-test-slave-qb36 krb5kdc[22488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:50:00 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:00 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:50:00.097993 23081 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:00.086515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.14.153.62:36287)
0619 16:50:00.086738 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:00.086745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:00.087146 (+ 401us) server_negotiation.cc:407] Connection header received
0619 16:50:00.087340 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:00.087346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:00.087429 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:00.087548 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:00.088963 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.090116 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.091488 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.092172 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.092303 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:00.097726 (+ 5423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41182: BlockingRecv error: recv got EOF from 127.0.0.1:41182 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:50:00 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM
I20260619 16:50:00.116748 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:00.101090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.14.153.62:36287)
0619 16:50:00.101255 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:00.101262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:00.101527 (+ 265us) server_negotiation.cc:407] Connection header received
0619 16:50:00.101728 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:00.101735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:00.101813 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:00.101922 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:00.103200 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.104308 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.105546 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.106181 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.106309 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:00.110203 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:00.110210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:00.110215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:00.110263 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:00.113741 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:00.114567 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:00.114573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:00.114577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:00.114755 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:00.115188 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:00.115193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:00.115197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:00.115550 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:00.115844 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:00.116132 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 16:50:00.116368 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:50:00.120744 22531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:00.121435 22531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:00.129748 22669 tablet_service.cc:1467] Processing CreateTablet for tablet 847d80dc69154aad9796f9f231307f46 (DEFAULT_TABLE table=test-table [id=aac814b4c8b1414885d498583c752318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:00.129979 22808 tablet_service.cc:1467] Processing CreateTablet for tablet 847d80dc69154aad9796f9f231307f46 (DEFAULT_TABLE table=test-table [id=aac814b4c8b1414885d498583c752318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:00.130239 22669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847d80dc69154aad9796f9f231307f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:00.130461 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847d80dc69154aad9796f9f231307f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:00.137069 23044 tablet_bootstrap.cc:492] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: Bootstrap starting.
I20260619 16:50:00.137701 23045 tablet_bootstrap.cc:492] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: Bootstrap starting.
I20260619 16:50:00.138707 23044 tablet_bootstrap.cc:654] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:00.139393 23045 tablet_bootstrap.cc:654] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:00.140583 22951 tablet_service.cc:1467] Processing CreateTablet for tablet 847d80dc69154aad9796f9f231307f46 (DEFAULT_TABLE table=test-table [id=aac814b4c8b1414885d498583c752318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:00.141003 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847d80dc69154aad9796f9f231307f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:00.142503 23044 tablet_bootstrap.cc:492] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: No bootstrap required, opened a new log
I20260619 16:50:00.142755 23044 ts_tablet_manager.cc:1397] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 16:50:00.143788 23044 raft_consensus.cc:348] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.144256 23044 raft_consensus.cc:374] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:00.144434 23044 raft_consensus.cc:729] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92093dd6b35417f8eee869cf3bc4aeb, State: Initialized, Role: FOLLOWER
I20260619 16:50:00.144819 23044 consensus_queue.cc:260] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.148077 23085 tablet_bootstrap.cc:492] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: Bootstrap starting.
I20260619 16:50:00.150099 23085 tablet_bootstrap.cc:654] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:00.150274 23044 ts_tablet_manager.cc:1428] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260619 16:50:00.150871 23045 tablet_bootstrap.cc:492] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: No bootstrap required, opened a new log
I20260619 16:50:00.151003 23045 ts_tablet_manager.cc:1397] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260619 16:50:00.151842 23045 raft_consensus.cc:348] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.152920 23045 raft_consensus.cc:374] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:00.153096 23085 tablet_bootstrap.cc:492] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: No bootstrap required, opened a new log
I20260619 16:50:00.153122 23045 raft_consensus.cc:729] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e96c2a1fcd04b248bfa867795679918, State: Initialized, Role: FOLLOWER
I20260619 16:50:00.153303 23085 ts_tablet_manager.cc:1397] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 16:50:00.153401 23045 consensus_queue.cc:260] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.153975 23045 ts_tablet_manager.cc:1428] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260619 16:50:00.154013 23085 raft_consensus.cc:348] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.154325 23085 raft_consensus.cc:374] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:00.154435 23085 raft_consensus.cc:729] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668a10e9b0214ac2a776638d52fdab9f, State: Initialized, Role: FOLLOWER
I20260619 16:50:00.154702 23085 consensus_queue.cc:260] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.155193 23085 ts_tablet_manager.cc:1428] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260619 16:50:00.182197 23108 raft_consensus.cc:482] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:00.182444 23108 raft_consensus.cc:504] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.183331 23108 leader_election.cc:288] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107)
I20260619 16:50:00.183917 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847d80dc69154aad9796f9f231307f46" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e96c2a1fcd04b248bfa867795679918" is_pre_election: true
I20260619 16:50:00.183944 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847d80dc69154aad9796f9f231307f46" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" is_pre_election: true
I20260619 16:50:00.184201 22689 raft_consensus.cc:2454] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 0.
I20260619 16:50:00.184216 22832 raft_consensus.cc:2454] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 0.
I20260619 16:50:00.184680 22902 leader_election.cc:302] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, 7e96c2a1fcd04b248bfa867795679918; no voters:
I20260619 16:50:00.184979 23108 raft_consensus.cc:2781] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:00.185106 23108 raft_consensus.cc:482] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:00.185209 23108 raft_consensus.cc:3037] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:00.188376 23108 raft_consensus.cc:504] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.189261 23108 leader_election.cc:288] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 election: Requested vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107)
I20260619 16:50:00.189571 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847d80dc69154aad9796f9f231307f46" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e96c2a1fcd04b248bfa867795679918"
I20260619 16:50:00.189585 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847d80dc69154aad9796f9f231307f46" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb"
I20260619 16:50:00.189791 22689 raft_consensus.cc:3037] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:00.189836 22832 raft_consensus.cc:3037] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:00.192137 22832 raft_consensus.cc:2454] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 1.
I20260619 16:50:00.192627 22902 leader_election.cc:302] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, 7e96c2a1fcd04b248bfa867795679918; no voters:
I20260619 16:50:00.192695 22689 raft_consensus.cc:2454] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 1.
I20260619 16:50:00.192931 23108 raft_consensus.cc:2781] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:00.193156 23108 raft_consensus.cc:686] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 1 LEADER]: Becoming Leader. State: Replica: 668a10e9b0214ac2a776638d52fdab9f, State: Running, Role: LEADER
I20260619 16:50:00.193756 23108 consensus_queue.cc:237] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.196363 22533 catalog_manager.cc:5495] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3). New cstate: current_term: 1 leader_uuid: "668a10e9b0214ac2a776638d52fdab9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } health_report { overall_health: HEALTHY } } }
I20260619 16:50:00.210897 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:00.205063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.14.153.3:39245)
0619 16:50:00.205296 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:00.205301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:00.205407 (+ 106us) server_negotiation.cc:407] Connection header received
0619 16:50:00.205631 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:00.205634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:00.205694 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:00.205767 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:00.206148 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.206905 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.208808 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.209239 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.209343 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:00.209916 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:00.210045 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:00.210124 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:00.210323 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 16:50:00.210396 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:50:00.213678 22689 raft_consensus.cc:1261] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Refusing update from remote peer 668a10e9b0214ac2a776638d52fdab9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:00.213848 22832 raft_consensus.cc:1261] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Refusing update from remote peer 668a10e9b0214ac2a776638d52fdab9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:00.214375 23108 consensus_queue.cc:1035] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:00.214789 23063 consensus_queue.cc:1035] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:00.231310 22532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:00.231514 22532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:00.234479 22532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76: Sending DeleteTablet for 3 replicas of tablet 847d80dc69154aad9796f9f231307f46
I20260619 16:50:00.235353 22669 tablet_service.cc:1514] Processing DeleteTablet for tablet 847d80dc69154aad9796f9f231307f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:00 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:33902
I20260619 16:50:00.235384 22808 tablet_service.cc:1514] Processing DeleteTablet for tablet 847d80dc69154aad9796f9f231307f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:00 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260619 16:50:00.235675 23113 tablet_replica.cc:330] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: stopping tablet replica
I20260619 16:50:00.235819 23114 tablet_replica.cc:330] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: stopping tablet replica
I20260619 16:50:00.235816 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 847d80dc69154aad9796f9f231307f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:00 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:36020
I20260619 16:50:00.235872 23113 raft_consensus.cc:2229] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:00.236016 23114 raft_consensus.cc:2229] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:00.236083 23113 raft_consensus.cc:2258] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:00.236245 23114 raft_consensus.cc:2258] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:00.236492 23112 tablet_replica.cc:330] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: stopping tablet replica
I20260619 16:50:00.236670 23112 raft_consensus.cc:2229] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:00.236929 23112 raft_consensus.cc:2258] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:00.237162 23113 ts_tablet_manager.cc:1905] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:00.237965 23112 ts_tablet_manager.cc:1905] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:00.242238 23112 ts_tablet_manager.cc:1918] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:00.242395 23112 log.cc:1196] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/847d80dc69154aad9796f9f231307f46
I20260619 16:50:00.242849 23112 ts_tablet_manager.cc:1939] T 847d80dc69154aad9796f9f231307f46 P 668a10e9b0214ac2a776638d52fdab9f: Deleting consensus metadata
I20260619 16:50:00.243067 23114 ts_tablet_manager.cc:1905] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:00.243805 23113 ts_tablet_manager.cc:1918] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:00.243988 23113 log.cc:1196] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/847d80dc69154aad9796f9f231307f46
I20260619 16:50:00.244120 22518 catalog_manager.cc:4841] TS 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3:39245): tablet 847d80dc69154aad9796f9f231307f46 (table test-table [id=aac814b4c8b1414885d498583c752318]) successfully deleted
I20260619 16:50:00.244447 23113 ts_tablet_manager.cc:1939] T 847d80dc69154aad9796f9f231307f46 P f92093dd6b35417f8eee869cf3bc4aeb: Deleting consensus metadata
I20260619 16:50:00.245463 22517 catalog_manager.cc:4841] TS f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983): tablet 847d80dc69154aad9796f9f231307f46 (table test-table [id=aac814b4c8b1414885d498583c752318]) successfully deleted
I20260619 16:50:00.248266 23114 ts_tablet_manager.cc:1918] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:00.248456 23114 log.cc:1196] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/847d80dc69154aad9796f9f231307f46
I20260619 16:50:00.249439 23114 ts_tablet_manager.cc:1939] T 847d80dc69154aad9796f9f231307f46 P 7e96c2a1fcd04b248bfa867795679918: Deleting consensus metadata
I20260619 16:50:00.249835 22532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:00.250777 22532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:00.250775 22520 catalog_manager.cc:4841] TS 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107): tablet 847d80dc69154aad9796f9f231307f46 (table test-table [id=aac814b4c8b1414885d498583c752318]) successfully deleted
I20260619 16:50:00.258610 22808 tablet_service.cc:1467] Processing CreateTablet for tablet a73b74afcfbc4f658c8d5608db3704ec (DEFAULT_TABLE table=test-table [id=68315eca2c264613b1aa2d09b582d9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:00.259075 22951 tablet_service.cc:1467] Processing CreateTablet for tablet a73b74afcfbc4f658c8d5608db3704ec (DEFAULT_TABLE table=test-table [id=68315eca2c264613b1aa2d09b582d9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:00.259202 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73b74afcfbc4f658c8d5608db3704ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:00.259508 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73b74afcfbc4f658c8d5608db3704ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:00.262598 22669 tablet_service.cc:1467] Processing CreateTablet for tablet a73b74afcfbc4f658c8d5608db3704ec (DEFAULT_TABLE table=test-table [id=68315eca2c264613b1aa2d09b582d9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:00.262976 22669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73b74afcfbc4f658c8d5608db3704ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:00.264972 23085 tablet_bootstrap.cc:492] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: Bootstrap starting.
I20260619 16:50:00.266871 23085 tablet_bootstrap.cc:654] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:00.267905 23045 tablet_bootstrap.cc:492] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: Bootstrap starting.
I20260619 16:50:00.270654 23044 tablet_bootstrap.cc:492] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: Bootstrap starting.
I20260619 16:50:00.270678 23045 tablet_bootstrap.cc:654] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:00.272719 23044 tablet_bootstrap.cc:654] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:00.278888 23085 tablet_bootstrap.cc:492] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: No bootstrap required, opened a new log
I20260619 16:50:00.279202 23085 ts_tablet_manager.cc:1397] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260619 16:50:00.280213 23085 raft_consensus.cc:348] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.280543 23085 raft_consensus.cc:374] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:00.280712 23085 raft_consensus.cc:729] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668a10e9b0214ac2a776638d52fdab9f, State: Initialized, Role: FOLLOWER
I20260619 16:50:00.281121 23085 consensus_queue.cc:260] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.282474 23045 tablet_bootstrap.cc:492] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: No bootstrap required, opened a new log
I20260619 16:50:00.282734 23045 ts_tablet_manager.cc:1397] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260619 16:50:00.283386 23044 tablet_bootstrap.cc:492] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: No bootstrap required, opened a new log
I20260619 16:50:00.283571 23044 ts_tablet_manager.cc:1397] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260619 16:50:00.284348 23044 raft_consensus.cc:348] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.284564 23085 ts_tablet_manager.cc:1428] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260619 16:50:00.284655 23044 raft_consensus.cc:374] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:00.284735 23044 raft_consensus.cc:729] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92093dd6b35417f8eee869cf3bc4aeb, State: Initialized, Role: FOLLOWER
I20260619 16:50:00.284996 23044 consensus_queue.cc:260] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.284996 23045 raft_consensus.cc:348] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.285284 23045 raft_consensus.cc:374] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:00.285387 23045 raft_consensus.cc:729] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e96c2a1fcd04b248bfa867795679918, State: Initialized, Role: FOLLOWER
I20260619 16:50:00.285642 23045 consensus_queue.cc:260] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.286183 23045 ts_tablet_manager.cc:1428] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260619 16:50:00.287602 23044 ts_tablet_manager.cc:1428] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:50:00.296499 23108 raft_consensus.cc:482] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:00.296715 23108 raft_consensus.cc:504] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.297415 23108 leader_election.cc:288] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107)
I20260619 16:50:00.297914 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73b74afcfbc4f658c8d5608db3704ec" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" is_pre_election: true
I20260619 16:50:00.297986 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73b74afcfbc4f658c8d5608db3704ec" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e96c2a1fcd04b248bfa867795679918" is_pre_election: true
I20260619 16:50:00.298171 22689 raft_consensus.cc:2454] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 0.
I20260619 16:50:00.298244 22832 raft_consensus.cc:2454] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 0.
I20260619 16:50:00.298628 22902 leader_election.cc:302] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, 7e96c2a1fcd04b248bfa867795679918; no voters:
I20260619 16:50:00.298971 23108 raft_consensus.cc:2781] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:00.299117 23108 raft_consensus.cc:482] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:00.299201 23108 raft_consensus.cc:3037] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:00.301578 23108 raft_consensus.cc:504] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.302253 23108 leader_election.cc:288] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 election: Requested vote from peers f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983), 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107)
I20260619 16:50:00.302711 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73b74afcfbc4f658c8d5608db3704ec" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92093dd6b35417f8eee869cf3bc4aeb"
I20260619 16:50:00.302752 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73b74afcfbc4f658c8d5608db3704ec" candidate_uuid: "668a10e9b0214ac2a776638d52fdab9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e96c2a1fcd04b248bfa867795679918"
I20260619 16:50:00.302915 22689 raft_consensus.cc:3037] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:00.303016 22832 raft_consensus.cc:3037] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:00.304925 22689 raft_consensus.cc:2454] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 1.
I20260619 16:50:00.305326 22899 leader_election.cc:302] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668a10e9b0214ac2a776638d52fdab9f, f92093dd6b35417f8eee869cf3bc4aeb; no voters:
I20260619 16:50:00.305603 23108 raft_consensus.cc:2781] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:00.305780 23108 raft_consensus.cc:686] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 1 LEADER]: Becoming Leader. State: Replica: 668a10e9b0214ac2a776638d52fdab9f, State: Running, Role: LEADER
I20260619 16:50:00.306034 23108 consensus_queue.cc:237] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } }
I20260619 16:50:00.306146 22832 raft_consensus.cc:2454] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668a10e9b0214ac2a776638d52fdab9f in term 1.
I20260619 16:50:00.308872 22531 catalog_manager.cc:5495] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3). New cstate: current_term: 1 leader_uuid: "668a10e9b0214ac2a776638d52fdab9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668a10e9b0214ac2a776638d52fdab9f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 39245 } health_report { overall_health: HEALTHY } } }
I20260619 16:50:00.344313 22689 raft_consensus.cc:1261] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Refusing update from remote peer 668a10e9b0214ac2a776638d52fdab9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:00.344347 22832 raft_consensus.cc:1261] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Refusing update from remote peer 668a10e9b0214ac2a776638d52fdab9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:00.344946 23063 consensus_queue.cc:1035] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92093dd6b35417f8eee869cf3bc4aeb" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:00.345196 23108 consensus_queue.cc:1035] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e96c2a1fcd04b248bfa867795679918" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:00.402945 22533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:00.403247 22533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:00.406034 22533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f34fc1d779b4f81b2f074cd8cf80a76: Sending DeleteTablet for 3 replicas of tablet a73b74afcfbc4f658c8d5608db3704ec
I20260619 16:50:00.406889 22669 tablet_service.cc:1514] Processing DeleteTablet for tablet a73b74afcfbc4f658c8d5608db3704ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:00 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:33902
I20260619 16:50:00.406920 22808 tablet_service.cc:1514] Processing DeleteTablet for tablet a73b74afcfbc4f658c8d5608db3704ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:00 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:40950
I20260619 16:50:00.407119 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet a73b74afcfbc4f658c8d5608db3704ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:00 UTC) from {username='oryx', principal='oryx/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:36020
I20260619 16:50:00.407225 23113 tablet_replica.cc:330] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: stopping tablet replica
I20260619 16:50:00.407366 23114 tablet_replica.cc:330] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: stopping tablet replica
I20260619 16:50:00.407389 23112 tablet_replica.cc:330] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: stopping tablet replica
I20260619 16:50:00.407425 23113 raft_consensus.cc:2229] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:00.407533 23114 raft_consensus.cc:2229] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:00.407550 23112 raft_consensus.cc:2229] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:00.407665 23113 raft_consensus.cc:2258] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:00.407724 23114 raft_consensus.cc:2258] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:00.407873 23112 raft_consensus.cc:2258] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:00.408624 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 22604
I20260619 16:50:00.408862 23113 ts_tablet_manager.cc:1905] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:00.408900 23114 ts_tablet_manager.cc:1905] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:00.411629 23112 ts_tablet_manager.cc:1905] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:00.415620 23113 ts_tablet_manager.cc:1918] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:50:00.415804 23113 log.cc:1196] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/a73b74afcfbc4f658c8d5608db3704ec
I20260619 16:50:00.416304 23113 ts_tablet_manager.cc:1939] T a73b74afcfbc4f658c8d5608db3704ec P f92093dd6b35417f8eee869cf3bc4aeb: Deleting consensus metadata
I20260619 16:50:00.416366 23114 ts_tablet_manager.cc:1918] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:50:00.416554 23114 log.cc:1196] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/a73b74afcfbc4f658c8d5608db3704ec
I20260619 16:50:00.416992 23114 ts_tablet_manager.cc:1939] T a73b74afcfbc4f658c8d5608db3704ec P 7e96c2a1fcd04b248bfa867795679918: Deleting consensus metadata
I20260619 16:50:00.417503 22517 catalog_manager.cc:4841] TS f92093dd6b35417f8eee869cf3bc4aeb (127.14.153.1:34983): tablet a73b74afcfbc4f658c8d5608db3704ec (table test-table [id=68315eca2c264613b1aa2d09b582d9ca]) successfully deleted
I20260619 16:50:00.418135 22520 catalog_manager.cc:4841] TS 7e96c2a1fcd04b248bfa867795679918 (127.14.153.2:44107): tablet a73b74afcfbc4f658c8d5608db3704ec (table test-table [id=68315eca2c264613b1aa2d09b582d9ca]) successfully deleted
I20260619 16:50:00.419710 23112 ts_tablet_manager.cc:1918] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:50:00.419866 23112 log.cc:1196] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/a73b74afcfbc4f658c8d5608db3704ec
I20260619 16:50:00.420305 23112 ts_tablet_manager.cc:1939] T a73b74afcfbc4f658c8d5608db3704ec P 668a10e9b0214ac2a776638d52fdab9f: Deleting consensus metadata
Jun 19 16:50:00 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.1@KRBTEST.COM
I20260619 16:50:00.421489 22518 catalog_manager.cc:4841] TS 668a10e9b0214ac2a776638d52fdab9f (127.14.153.3:39245): tablet a73b74afcfbc4f658c8d5608db3704ec (table test-table [id=68315eca2c264613b1aa2d09b582d9ca]) successfully deleted
I20260619 16:50:00.428181 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:00.409958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.14.153.1:34983)
0619 16:50:00.411313 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:00.411318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:00.411332 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:50:00.411376 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:00.411379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:00.411443 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:00.411562 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:00.413140 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.414435 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.416433 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.417187 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.417363 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:00.422437 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:00.422453 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:00.422471 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:00.422508 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:00.425459 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:00.426236 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:00.426244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:00.426246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:00.426392 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:00.426747 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:00.426751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:00.426753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:00.427221 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:00.427452 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:00.427713 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 16:50:00.427895 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1247,"thread_start_us":1140,"threads_started":1}
I20260619 16:50:00.428907 22735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:00.526026 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 22604
I20260619 16:50:00.544656 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 22744
Jun 19 16:50:00 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.2@KRBTEST.COM
I20260619 16:50:00.559831 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:00.545442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40978 (local address 127.14.153.2:44107)
0619 16:50:00.545959 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:00.545964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:00.545980 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:00.546034 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:00.546038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:00.546094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:00.546193 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:00.547410 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.548099 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.549233 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.549616 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.549707 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:00.553850 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:00.553866 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:00.553884 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:00.553920 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:00.556872 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:00.557625 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:00.557637 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:00.557640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:00.557813 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:00.558258 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:00.558265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:00.558270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:00.558661 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:00.558886 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:00.559260 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 16:50:00.559454 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":312,"threads_started":1}
I20260619 16:50:00.560513 22874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:00.683429 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 22744
I20260619 16:50:00.707413 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 22883
Jun 19 16:50:00 dist-test-slave-qb36 krb5kdc[22488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.153.3@KRBTEST.COM
I20260619 16:50:00.722033 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:00.708144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.14.153.3:39245)
0619 16:50:00.708311 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:00.708315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:00.708335 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:50:00.708481 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:00.708485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:00.708542 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:00.708635 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:00.709859 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.710550 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.711811 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.712212 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.712305 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:00.716481 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:00.716496 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:00.716513 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:00.716548 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:00.719517 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:00.720235 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:00.720243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:00.720245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:00.720382 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:00.720791 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:00.720795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:00.720797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:00.721170 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:00.721404 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:00.721628 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 16:50:00.721797 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:50:00.722685 23013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:00.847692 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 22883
I20260619 16:50:00.869724 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 22501
I20260619 16:50:00.870884 22573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:00.986490 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 22501
2026-06-19T16:50:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6630 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:01 dist-test-slave-qb36 krb5kdc[23149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:01 dist-test-slave-qb36 krb5kdc[23149](info): set up 2 sockets
Jun 19 16:50:01 dist-test-slave-qb36 krb5kdc[23149](info): commencing operation
krb5kdc: starting...
W20260619 16:50:02.104149 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:50:02 dist-test-slave-qb36 krb5kdc[23149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:50:02.326306 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34487
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:02.650429 23162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:02.650792 23162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:02.650960 23162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:02.658392 23162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:02.658546 23162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:02.658659 23162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:02.658762 23162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:02.658852 23162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:02.669025 23162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:02.670400 23162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:02.671800 23162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:02.683419 23167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:02.683398 23168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:02.683624 23170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:02.684173 23162 server_base.cc:1034] running on GCE node
I20260619 16:50:02.684980 23162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:02.686530 23162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:02.687764 23162 hybrid_clock.cc:648] HybridClock initialized: now 1781887802687724 us; error 45 us; skew 500 ppm
I20260619 16:50:02.688122 23162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T16:50:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1765 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:02 dist-test-slave-qb36 krb5kdc[23176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:02 dist-test-slave-qb36 krb5kdc[23176](info): set up 2 sockets
Jun 19 16:50:02 dist-test-slave-qb36 krb5kdc[23176](info): commencing operation
krb5kdc: starting...
W20260619 16:50:03.864156 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:50:04 dist-test-slave-qb36 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:50:04.090981 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:36833
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:36833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:04.418635 23189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:04.418951 23189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:04.419160 23189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:04.426560 23189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:04.426676 23189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:04.426781 23189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:04.426863 23189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:04.426926 23189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:04.437203 23189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:36833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:36833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:04.438692 23189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:04.440127 23189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:04.452162 23194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:04.452280 23195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:04.452343 23197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:04.453548 23189 server_base.cc:1034] running on GCE node
I20260619 16:50:04.454327 23189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:04.455821 23189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:04.457063 23189 hybrid_clock.cc:648] HybridClock initialized: now 1781887804457018 us; error 50 us; skew 500 ppm
I20260619 16:50:04.457408 23189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:04 dist-test-slave-qb36 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887804, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:04.464192 23189 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:50:04.465785 23189 webserver.cc:460] Webserver started at http://127.14.153.62:33591/ using document root <none> and password file <none>
I20260619 16:50:04.466504 23189 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:04.466652 23189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:04.466970 23189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:04.470053 23189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "a1256efbabf34cea8a878fa91cb9834e"
format_stamp: "Formatted at 2026-06-19 16:50:04 on dist-test-slave-qb36"
server_key: "f2adb82b4936ad55ad4395ce91771038"
server_key_iv: "44df1de07ddf41dd861fa65832d8e9c8"
server_key_version: "encryptionkey@0"
I20260619 16:50:04.470831 23189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "a1256efbabf34cea8a878fa91cb9834e"
format_stamp: "Formatted at 2026-06-19 16:50:04 on dist-test-slave-qb36"
server_key: "f2adb82b4936ad55ad4395ce91771038"
server_key_iv: "44df1de07ddf41dd861fa65832d8e9c8"
server_key_version: "encryptionkey@0"
I20260619 16:50:04.476408 23189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:50:04.480166 23204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:04.481658 23189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:50:04.481861 23189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "a1256efbabf34cea8a878fa91cb9834e"
format_stamp: "Formatted at 2026-06-19 16:50:04 on dist-test-slave-qb36"
server_key: "f2adb82b4936ad55ad4395ce91771038"
server_key_iv: "44df1de07ddf41dd861fa65832d8e9c8"
server_key_version: "encryptionkey@0"
I20260619 16:50:04.482082 23189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:04.526191 23189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:04.530470 23189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:04.530783 23189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:04.549628 23189 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:36833
I20260619 16:50:04.549635 23255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:36833 every 8 connection(s)
I20260619 16:50:04.551074 23189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:04.553210 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 23189
I20260619 16:50:04.553412 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:50:04.554206 14948 external_mini_cluster.cc:1428] Setting key d8879201631c877f8769bfe4bb5d3a12
I20260619 16:50:04.555709 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:50:04 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:50:04.569938 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: Bootstrap starting.
I20260619 16:50:04.573971 23256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:04.575289 23256 log.cc:824] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:04.578819 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: No bootstrap required, opened a new log
I20260619 16:50:04.578966 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:04.556312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.14.153.62:36833)
0619 16:50:04.557078 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:04.557090 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:04.557133 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:50:04.558725 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:04.558769 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:04.559247 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:04.559586 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:04.561428 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:04.562822 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:04.564636 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:04.565327 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:04.565494 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:04.570217 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:04.570225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:04.570245 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:04.570287 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:04.574215 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:04.574935 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:04.574944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:04.574953 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:04.575189 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:04.575665 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:04.575670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:04.575674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:04.576351 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:04.576563 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:04.577136 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 16:50:04.577577 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":378,"threads_started":1}
I20260619 16:50:04.586884 23256 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } }
I20260619 16:50:04.587409 23256 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:04.587543 23256 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1256efbabf34cea8a878fa91cb9834e, State: Initialized, Role: FOLLOWER
I20260619 16:50:04.588096 23256 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } }
I20260619 16:50:04.588323 23256 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:04.588454 23256 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:04.588661 23256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:04.590819 23256 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } }
I20260619 16:50:04.591385 23256 leader_election.cc:302] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1256efbabf34cea8a878fa91cb9834e; no voters:
I20260619 16:50:04.592072 23256 leader_election.cc:288] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:04.592219 23261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:04.592901 23261 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 1 LEADER]: Becoming Leader. State: Replica: a1256efbabf34cea8a878fa91cb9834e, State: Running, Role: LEADER
I20260619 16:50:04.593560 23261 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } }
I20260619 16:50:04.594331 23256 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:04.596486 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1256efbabf34cea8a878fa91cb9834e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } } }
I20260619 16:50:04.596933 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:04.596508 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1256efbabf34cea8a878fa91cb9834e. Latest consensus state: current_term: 1 leader_uuid: "a1256efbabf34cea8a878fa91cb9834e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } } }
I20260619 16:50:04.597746 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:04.598670 23267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:04.605214 23267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:04.614589 23267 catalog_manager.cc:1347] Generated new cluster ID: 5876fc2c9c364d99a49b256c25e8b7dd
I20260619 16:50:04.614727 23267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:04.627485 23267 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:04.629401 23267 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:04.641925 23267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: Generated new TSK 0
I20260619 16:50:04.642836 23267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:50:04.714363 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:36833
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:05.063892 23282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:05.064198 23282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:05.064352 23282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:05.071946 23282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:05.072069 23282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:05.072237 23282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:50:05.082859 23282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:05.084249 23282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:05.085721 23282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:05.099345 23287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:05.099431 23288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:05.099453 23290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:05.100788 23282 server_base.cc:1034] running on GCE node
I20260619 16:50:05.101440 23282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:05.102459 23282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:05.103693 23282 hybrid_clock.cc:648] HybridClock initialized: now 1781887805103649 us; error 40 us; skew 500 ppm
I20260619 16:50:05.104020 23282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:05 dist-test-slave-qb36 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887805, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:05.107383 23282 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:50:05.108810 23282 webserver.cc:460] Webserver started at http://127.14.153.1:40987/ using document root <none> and password file <none>
I20260619 16:50:05.109500 23282 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:05.109603 23282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:05.109876 23282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:05.112986 23282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "576f7bd68a264c3aad4e2bb05b47d10f"
format_stamp: "Formatted at 2026-06-19 16:50:05 on dist-test-slave-qb36"
server_key: "1c5cd72073456c16fd9d2e3cebf1c7f5"
server_key_iv: "2616c849884d03bb7cf69d2d9fdcfc3b"
server_key_version: "encryptionkey@0"
I20260619 16:50:05.113750 23282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "576f7bd68a264c3aad4e2bb05b47d10f"
format_stamp: "Formatted at 2026-06-19 16:50:05 on dist-test-slave-qb36"
server_key: "1c5cd72073456c16fd9d2e3cebf1c7f5"
server_key_iv: "2616c849884d03bb7cf69d2d9fdcfc3b"
server_key_version: "encryptionkey@0"
I20260619 16:50:05.119239 23282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 16:50:05.122668 23297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:05.124089 23282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:05.124275 23282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "576f7bd68a264c3aad4e2bb05b47d10f"
format_stamp: "Formatted at 2026-06-19 16:50:05 on dist-test-slave-qb36"
server_key: "1c5cd72073456c16fd9d2e3cebf1c7f5"
server_key_iv: "2616c849884d03bb7cf69d2d9fdcfc3b"
server_key_version: "encryptionkey@0"
I20260619 16:50:05.124473 23282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:05.141541 23282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:05.145816 23282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:05.146113 23282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:05.147145 23282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:05.148496 23282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:05.148603 23282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:05.148803 23282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:05.148904 23282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:05.181278 23282 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:33023
I20260619 16:50:05.181367 23409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:33023 every 8 connection(s)
I20260619 16:50:05.182571 23282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:05.190045 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 23282
I20260619 16:50:05.190248 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:05.191097 14948 external_mini_cluster.cc:1428] Setting key 3676fd0a596f463cd7b70416c1dbeddf
Jun 19 16:50:05 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887805, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:50:05.205641 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:05.185276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:39411 (local address 127.14.153.62:36833)
0619 16:50:05.185783 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:05.185789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:05.186819 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 16:50:05.188732 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:05.188736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:05.188801 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:05.188898 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:05.191079 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:05.191978 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:05.194133 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:05.194584 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:05.194684 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:05.199380 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:05.199388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:05.199395 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:05.199438 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:05.202264 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:05.203251 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:05.203256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:05.203259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:05.203387 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:05.203874 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:05.203878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:05.203880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:05.204154 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:05.204262 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:05.205206 (+ 944us) server_negotiation.cc:299] Negotiation successful
0619 16:50:05.205375 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260619 16:50:05.207213 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:05.185819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:39411)
0619 16:50:05.186582 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:05.186699 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:05.188453 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:05.189166 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:05.189174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:05.189879 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:05.190770 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:05.190809 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:05.192165 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:05.192172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:05.193972 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:05.193990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:05.194838 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:05.194848 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:05.195222 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:05.196055 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:05.196090 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:05.199101 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:05.202411 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:05.202423 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:05.202445 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:05.203126 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:05.203507 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:05.203512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:05.203518 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:05.203763 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:05.204279 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:05.204291 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:05.204779 (+ 488us) client_negotiation.cc:769] Sending connection context
0619 16:50:05.205089 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 16:50:05.205668 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":271,"threads_started":1}
I20260619 16:50:05.208873 23410 heartbeater.cc:347] Connected to a master server at 127.14.153.62:36833
I20260619 16:50:05.209210 23410 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:05.209976 23410 heartbeater.cc:511] Master 127.14.153.62:36833 requested a full tablet report, sending...
I20260619 16:50:05.212212 23221 ts_manager.cc:194] Registered new tserver with Master: 576f7bd68a264c3aad4e2bb05b47d10f (127.14.153.1:33023)
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:50:05.213726 23221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:39411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:50:05.243165 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:36833
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:05.572371 23416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:05.572695 23416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:05.572851 23416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:05.580144 23416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:05.580307 23416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:05.580493 23416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:50:05.590492 23416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:05.591837 23416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:05.593153 23416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:05.606240 23422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:05.606271 23421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:05.606580 23424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:05.606915 23416 server_base.cc:1034] running on GCE node
I20260619 16:50:05.607663 23416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:05.608568 23416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:05.609795 23416 hybrid_clock.cc:648] HybridClock initialized: now 1781887805609733 us; error 63 us; skew 500 ppm
I20260619 16:50:05.610116 23416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:05 dist-test-slave-qb36 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887805, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:05.613582 23416 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:50:05.615270 23416 webserver.cc:460] Webserver started at http://127.14.153.2:36541/ using document root <none> and password file <none>
I20260619 16:50:05.615989 23416 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:05.616133 23416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:05.616459 23416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:05.619516 23416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "028c2901692f4bf79de51e2aa96b9d74"
format_stamp: "Formatted at 2026-06-19 16:50:05 on dist-test-slave-qb36"
server_key: "f9e1213347e6be666baa42df82205c10"
server_key_iv: "83d458d4b4f0a810814b2fca57d6da7f"
server_key_version: "encryptionkey@0"
I20260619 16:50:05.620293 23416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "028c2901692f4bf79de51e2aa96b9d74"
format_stamp: "Formatted at 2026-06-19 16:50:05 on dist-test-slave-qb36"
server_key: "f9e1213347e6be666baa42df82205c10"
server_key_iv: "83d458d4b4f0a810814b2fca57d6da7f"
server_key_version: "encryptionkey@0"
I20260619 16:50:05.625537 23416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:50:05.629223 23431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:05.630612 23416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:05.630795 23416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "028c2901692f4bf79de51e2aa96b9d74"
format_stamp: "Formatted at 2026-06-19 16:50:05 on dist-test-slave-qb36"
server_key: "f9e1213347e6be666baa42df82205c10"
server_key_iv: "83d458d4b4f0a810814b2fca57d6da7f"
server_key_version: "encryptionkey@0"
I20260619 16:50:05.631009 23416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:05.650662 23416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:05.655131 23416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:05.655437 23416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:05.656458 23416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:05.657886 23416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:05.657960 23416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:05.658087 23416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:05.658149 23416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:05.692137 23416 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:37731
I20260619 16:50:05.692212 23543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:37731 every 8 connection(s)
I20260619 16:50:05.693414 23416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:05.694923 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 23416
I20260619 16:50:05.695118 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:05.695544 14948 external_mini_cluster.cc:1428] Setting key d3cb0b196dcc944c418068f5a80a763a
Jun 19 16:50:05 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887805, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:50:05.715481 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:05.696626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:34453 (local address 127.14.153.62:36833)
0619 16:50:05.696790 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:05.696795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:05.698136 (+ 1341us) server_negotiation.cc:407] Connection header received
0619 16:50:05.700405 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:05.700411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:05.700570 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:05.700683 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:05.702298 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:05.703017 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:05.704608 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:05.705043 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:05.705178 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:05.709093 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:05.709102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:05.709107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:05.709147 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:05.712310 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:05.713068 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:05.713074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:05.713078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:05.713223 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:05.713687 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:05.713693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:05.713697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:05.713978 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:05.714134 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:05.714854 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 16:50:05.715113 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:50:05.716598 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:05.697136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:34453)
0619 16:50:05.697949 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:05.698007 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:05.700038 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:05.700934 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:05.700943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:05.701567 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:05.702136 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:05.702154 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:05.703211 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:05.703217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:05.704447 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:05.704459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:05.705264 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:05.705271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:05.705500 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:05.706205 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:05.706231 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:05.708729 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:05.712461 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:05.712471 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:05.712490 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:05.712931 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:05.713338 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:05.713343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:05.713348 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:05.713571 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:05.714124 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:05.714133 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:05.714496 (+ 363us) client_negotiation.cc:769] Sending connection context
0619 16:50:05.714779 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 16:50:05.715376 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":355,"threads_started":1}
I20260619 16:50:05.718109 23544 heartbeater.cc:347] Connected to a master server at 127.14.153.62:36833
I20260619 16:50:05.718520 23544 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
I20260619 16:50:05.719375 23544 heartbeater.cc:511] Master 127.14.153.62:36833 requested a full tablet report, sending...
I20260619 16:50:05.720983 23221 ts_manager.cc:194] Registered new tserver with Master: 028c2901692f4bf79de51e2aa96b9d74 (127.14.153.2:37731)
I20260619 16:50:05.722221 23221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:34453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:50:05.746208 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:36833
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:06.088546 23549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:06.088840 23549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:06.088934 23549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:06.096485 23549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:06.096597 23549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:06.096710 23549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:50:06.106563 23549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:06.107887 23549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:06.109140 23549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:06.123056 23554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:06.122979 23555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:06.122943 23557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:06.124812 23549 server_base.cc:1034] running on GCE node
I20260619 16:50:06.125499 23549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:06.126533 23549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:06.127780 23549 hybrid_clock.cc:648] HybridClock initialized: now 1781887806127740 us; error 31 us; skew 500 ppm
I20260619 16:50:06.128158 23549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:06 dist-test-slave-qb36 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887806, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:06.131656 23549 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:50:06.133162 23549 webserver.cc:460] Webserver started at http://127.14.153.3:42437/ using document root <none> and password file <none>
I20260619 16:50:06.133917 23549 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:06.134074 23549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:06.134469 23549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:06.137655 23549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "a065eee9cdf94d3ab166f363ca1d52b0"
format_stamp: "Formatted at 2026-06-19 16:50:06 on dist-test-slave-qb36"
server_key: "b817eb734c93e3ff04b2978ada1daccd"
server_key_iv: "e54b32fbbe8bce7e86c99a7fdbf17721"
server_key_version: "encryptionkey@0"
I20260619 16:50:06.138474 23549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "a065eee9cdf94d3ab166f363ca1d52b0"
format_stamp: "Formatted at 2026-06-19 16:50:06 on dist-test-slave-qb36"
server_key: "b817eb734c93e3ff04b2978ada1daccd"
server_key_iv: "e54b32fbbe8bce7e86c99a7fdbf17721"
server_key_version: "encryptionkey@0"
I20260619 16:50:06.144135 23549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260619 16:50:06.147775 23564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:06.149088 23549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:50:06.149261 23549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "a065eee9cdf94d3ab166f363ca1d52b0"
format_stamp: "Formatted at 2026-06-19 16:50:06 on dist-test-slave-qb36"
server_key: "b817eb734c93e3ff04b2978ada1daccd"
server_key_iv: "e54b32fbbe8bce7e86c99a7fdbf17721"
server_key_version: "encryptionkey@0"
I20260619 16:50:06.149467 23549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:06.174218 23549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:06.178457 23549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:06.178772 23549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:06.179811 23549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:06.181483 23549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:06.181579 23549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:06.181694 23549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:06.181762 23549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:06.216374 23549 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:41593
I20260619 16:50:06.216434 23676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:41593 every 8 connection(s)
I20260619 16:50:06.217024 23410 heartbeater.cc:503] Master 127.14.153.62:36833 was elected leader, sending a full tablet report...
I20260619 16:50:06.218240 23549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:06.227581 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 23549
I20260619 16:50:06.227780 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:06.228236 14948 external_mini_cluster.cc:1428] Setting key 923dc15966b9c9d52e98bda0f03786e7
Jun 19 16:50:06 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887806, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:50:06.239216 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:06.220853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:49341 (local address 127.14.153.62:36833)
0619 16:50:06.221334 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:06.221338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:06.222193 (+ 855us) server_negotiation.cc:407] Connection header received
0619 16:50:06.224112 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:06.224118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:06.224211 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:06.224330 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:06.225914 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.226684 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.228634 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.229204 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.229397 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:06.233290 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:06.233295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:06.233299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:06.233331 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:06.236051 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:06.236823 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:06.236834 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:06.236836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:06.236954 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:06.237407 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:06.237412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:06.237415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:06.237699 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:06.237854 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:06.238565 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 16:50:06.238778 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260619 16:50:06.240196 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:06.221355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:49341)
0619 16:50:06.222030 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:06.222075 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:06.223830 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:06.224619 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:06.224629 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:06.225142 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:06.225738 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:06.225756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.226857 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.226861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:06.228463 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:06.228476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.229493 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.229503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:06.229828 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:06.230621 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:06.230643 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:06.233075 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:06.236212 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:06.236221 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:06.236240 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:06.236690 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:06.237082 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:06.237087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:06.237091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:06.237282 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:06.237831 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:06.237839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:06.238194 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 16:50:06.238465 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 16:50:06.239021 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":258,"threads_started":1}
I20260619 16:50:06.241875 23677 heartbeater.cc:347] Connected to a master server at 127.14.153.62:36833
I20260619 16:50:06.242244 23677 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:06.243136 23677 heartbeater.cc:511] Master 127.14.153.62:36833 requested a full tablet report, sending...
I20260619 16:50:06.244648 23221 ts_manager.cc:194] Registered new tserver with Master: a065eee9cdf94d3ab166f363ca1d52b0 (127.14.153.3:41593)
I20260619 16:50:06.245426 23221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:49341
I20260619 16:50:06.256301 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab.
Entry for principal oryx/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab.
WARNING: no policy specified for oryx/127.14.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab.
Entry for principal oryx/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab.
WARNING: no policy specified for oryx/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab.
Entry for principal oryx/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab.
I20260619 16:50:06.394663 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 23282
Jun 19 16:50:06 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:50:06.411779 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:06.395905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.14.153.1:33023)
0619 16:50:06.396517 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:06.396525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:06.396554 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:50:06.396616 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:06.396622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:06.396849 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:06.396987 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:06.398524 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.399628 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.401026 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.401783 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.401889 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:06.405918 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:06.405929 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:06.405944 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:06.405983 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:06.408877 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:06.409617 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:06.409625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:06.409627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:06.409762 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:06.410135 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:06.410143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:06.410147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:06.410486 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:06.410805 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:06.411132 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 16:50:06.411404 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":314,"threads_started":1}
I20260619 16:50:06.413074 23405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:06.496482 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 23282
I20260619 16:50:06.513182 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 23416
Jun 19 16:50:06 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:50:06.528843 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:06.514158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.14.153.2:37731)
0619 16:50:06.514704 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:06.514712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:06.514737 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:50:06.514795 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:06.514802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:06.515003 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:06.515217 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:06.516486 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.517517 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.518735 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.519462 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.519566 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:06.523550 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:06.523564 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:06.523580 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:06.523617 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:06.526406 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:06.526980 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:06.526988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:06.526990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:06.527176 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:06.527477 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:06.527480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:06.527482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:06.527810 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:06.528015 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:06.528253 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 16:50:06.528445 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":268,"threads_started":1}
I20260619 16:50:06.530104 23539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:06.611858 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 23416
I20260619 16:50:06.627949 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 23549
Jun 19 16:50:06 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:50:06.644718 23692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:06.628909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.14.153.3:41593)
0619 16:50:06.629594 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:06.629601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:06.629633 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:50:06.629698 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:06.629705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:06.629933 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:06.630074 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:06.631424 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.632484 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.633674 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:06.634237 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:06.634336 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:06.638639 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:06.638653 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:06.638669 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:06.638711 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:06.641650 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:06.642396 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:06.642404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:06.642406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:06.642555 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:06.642936 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:06.642940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:06.642942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:06.643493 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:06.643776 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:06.644054 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 16:50:06.644264 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":371,"threads_started":1}
I20260619 16:50:06.646128 23672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:06.730049 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 23549
I20260619 16:50:06.745695 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 23189
I20260619 16:50:06.746958 23251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:06.848819 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 23189
I20260619 16:50:06.869321 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:36833
--webserver_interface=127.14.153.62
--webserver_port=33591
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:36833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:07.194681 23695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:07.195039 23695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:07.195236 23695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:07.202553 23695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:07.202711 23695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:07.202823 23695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:07.202930 23695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:07.203015 23695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:07.213030 23695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:36833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:36833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=33591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:07.214613 23695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:07.216236 23695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:07.227941 23701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:07.227975 23703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:07.227941 23700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:07.228929 23695 server_base.cc:1034] running on GCE node
I20260619 16:50:07.229705 23695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:07.231230 23695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:07.232460 23695 hybrid_clock.cc:648] HybridClock initialized: now 1781887807232414 us; error 54 us; skew 500 ppm
I20260619 16:50:07.232806 23695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:07 dist-test-slave-qb36 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887807, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:07.235874 23695 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:50:07.237416 23695 webserver.cc:460] Webserver started at http://127.14.153.62:33591/ using document root <none> and password file <none>
I20260619 16:50:07.238104 23695 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:07.238266 23695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:07.243757 23695 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:50:07.246534 23710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:07.247975 23695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:50:07.248168 23695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "a1256efbabf34cea8a878fa91cb9834e"
format_stamp: "Formatted at 2026-06-19 16:50:04 on dist-test-slave-qb36"
server_key: "f2adb82b4936ad55ad4395ce91771038"
server_key_iv: "44df1de07ddf41dd861fa65832d8e9c8"
server_key_version: "encryptionkey@0"
I20260619 16:50:07.249147 23695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:07.280802 23695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:07.285250 23695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:07.285532 23695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:07.304210 23695 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:36833
I20260619 16:50:07.304229 23761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:36833 every 8 connection(s)
I20260619 16:50:07.305753 23695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:07.310948 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 23695
I20260619 16:50:07.311846 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:33023
--local_ip_for_outbound_sockets=127.14.153.1
--tserver_master_addrs=127.14.153.62:36833
--webserver_port=40987
--webserver_interface=127.14.153.1
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.153.1
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:50:07.312731 23762 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:50:07.315974 23762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: Bootstrap starting.
I20260619 16:50:07.342504 23762 log.cc:824] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:07.353614 23762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:50:07.354249 23762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: Bootstrap complete.
I20260619 16:50:07.367404 23762 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } }
I20260619 16:50:07.368604 23762 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1256efbabf34cea8a878fa91cb9834e, State: Initialized, Role: FOLLOWER
I20260619 16:50:07.369417 23762 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } }
I20260619 16:50:07.369740 23762 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:07.369858 23762 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:07.370052 23762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:50:07.375983 23762 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } }
I20260619 16:50:07.376776 23762 leader_election.cc:302] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1256efbabf34cea8a878fa91cb9834e; no voters:
I20260619 16:50:07.377564 23762 leader_election.cc:288] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:50:07.377725 23766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:50:07.379300 23766 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [term 2 LEADER]: Becoming Leader. State: Replica: a1256efbabf34cea8a878fa91cb9834e, State: Running, Role: LEADER
I20260619 16:50:07.379918 23766 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } }
I20260619 16:50:07.380123 23762 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:07.382319 23768 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1256efbabf34cea8a878fa91cb9834e. Latest consensus state: current_term: 2 leader_uuid: "a1256efbabf34cea8a878fa91cb9834e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } } }
I20260619 16:50:07.382707 23768 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:07.383265 23767 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1256efbabf34cea8a878fa91cb9834e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1256efbabf34cea8a878fa91cb9834e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 36833 } } }
I20260619 16:50:07.383591 23767 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:07.385291 23772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:07.390872 23772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:07.392289 23772 catalog_manager.cc:1259] Loaded cluster ID: 5876fc2c9c364d99a49b256c25e8b7dd
I20260619 16:50:07.392398 23772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:07.396453 23772 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:07.397933 23772 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1256efbabf34cea8a878fa91cb9834e: Loaded TSK: 0
I20260619 16:50:07.399814 23772 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:50:07.675539 23764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:07.675877 23764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:07.676051 23764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:07.683812 23764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:07.683979 23764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:07.684158 23764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:50:07.694234 23764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.1.keytab
--principal=oryx/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=40987
--tserver_master_addrs=127.14.153.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:07.695714 23764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:07.697085 23764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:07.710609 23788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:07.710613 23789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:07.710644 23791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:07.712232 23764 server_base.cc:1034] running on GCE node
I20260619 16:50:07.712909 23764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:07.714013 23764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:07.715260 23764 hybrid_clock.cc:648] HybridClock initialized: now 1781887807715179 us; error 50 us; skew 500 ppm
I20260619 16:50:07.715703 23764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:07 dist-test-slave-qb36 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887807, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:07.719906 23764 init.cc:377] Logged in from keytab as oryx/127.14.153.1@KRBTEST.COM (short username oryx)
I20260619 16:50:07.721621 23764 webserver.cc:460] Webserver started at http://127.14.153.1:40987/ using document root <none> and password file <none>
I20260619 16:50:07.722361 23764 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:07.722527 23764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:07.728307 23764 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:50:07.731315 23798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:07.732879 23764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:50:07.733072 23764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "576f7bd68a264c3aad4e2bb05b47d10f"
format_stamp: "Formatted at 2026-06-19 16:50:05 on dist-test-slave-qb36"
server_key: "1c5cd72073456c16fd9d2e3cebf1c7f5"
server_key_iv: "2616c849884d03bb7cf69d2d9fdcfc3b"
server_key_version: "encryptionkey@0"
I20260619 16:50:07.734048 23764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:07.747628 23764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:07.751979 23764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:07.752314 23764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:07.753346 23764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:07.754657 23764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:07.754747 23764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:07.754901 23764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:07.754995 23764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:07.794379 23764 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:33023
I20260619 16:50:07.794503 23910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:33023 every 8 connection(s)
I20260619 16:50:07.795967 23764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:07.797873 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 23764
I20260619 16:50:07.798532 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:37731
--local_ip_for_outbound_sockets=127.14.153.2
--tserver_master_addrs=127.14.153.62:36833
--webserver_port=36541
--webserver_interface=127.14.153.2
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.153.2
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:50:07 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:07 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:50:08.142288 23915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:08.142629 23915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:08.142784 23915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:08.150048 23915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:08.150211 23915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:08.150393 23915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:50:08.160534 23915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.2.keytab
--principal=oryx/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:37731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=36541
--tserver_master_addrs=127.14.153.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:08.161950 23915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:08.163295 23915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:08.177362 23921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:08.177974 23924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:08.178437 23922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:08.178813 23915 server_base.cc:1034] running on GCE node
I20260619 16:50:08.180027 23915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:08.181803 23915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:08.183118 23915 hybrid_clock.cc:648] HybridClock initialized: now 1781887808183025 us; error 62 us; skew 500 ppm
I20260619 16:50:08.183678 23915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:08 dist-test-slave-qb36 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887808, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:08.189319 23915 init.cc:377] Logged in from keytab as oryx/127.14.153.2@KRBTEST.COM (short username oryx)
I20260619 16:50:08.191851 23915 webserver.cc:460] Webserver started at http://127.14.153.2:36541/ using document root <none> and password file <none>
I20260619 16:50:08.193202 23915 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:08.193414 23915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:08.204108 23915 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260619 16:50:08.208935 23931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:08.211364 23915 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:50:08.211637 23915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "028c2901692f4bf79de51e2aa96b9d74"
format_stamp: "Formatted at 2026-06-19 16:50:05 on dist-test-slave-qb36"
server_key: "f9e1213347e6be666baa42df82205c10"
server_key_iv: "83d458d4b4f0a810814b2fca57d6da7f"
server_key_version: "encryptionkey@0"
I20260619 16:50:08.213680 23915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:08.248721 23915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:08.256685 23915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:08.257129 23915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:08.258764 23915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:08.261173 23915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:08.261333 23915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:08.261489 23915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:08.261596 23915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:08.301067 23915 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:37731
I20260619 16:50:08.301097 24043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:37731 every 8 connection(s)
I20260619 16:50:08.303861 23915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:08.311511 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 23915
I20260619 16:50:08.312229 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:41593
--local_ip_for_outbound_sockets=127.14.153.3
--tserver_master_addrs=127.14.153.62:36833
--webserver_port=42437
--webserver_interface=127.14.153.3
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.153.3
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:50:08 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:08 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:50:08.666327 24048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:08.666674 24048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:08.666847 24048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:08.674341 24048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:08.674585 24048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:08.674778 24048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:50:08.685480 24048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/krb5kdc/oryx_127.14.153.3.keytab
--principal=oryx/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:41593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=42437
--tserver_master_addrs=127.14.153.62:36833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:08.686930 24048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:08.688356 24048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:08.705864 24054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:08.707548 24055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:08.716784 24048 server_base.cc:1034] running on GCE node
W20260619 16:50:08.717964 24057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:08.718876 24048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:08.722931 24048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:08.724220 24048 hybrid_clock.cc:648] HybridClock initialized: now 1781887808724152 us; error 80 us; skew 500 ppm
I20260619 16:50:08.724620 24048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:08 dist-test-slave-qb36 krb5kdc[23176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887808, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:08.728185 24048 init.cc:377] Logged in from keytab as oryx/127.14.153.3@KRBTEST.COM (short username oryx)
I20260619 16:50:08.732399 24048 webserver.cc:460] Webserver started at http://127.14.153.3:42437/ using document root <none> and password file <none>
I20260619 16:50:08.733166 24048 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:08.733332 24048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:08.739660 24048 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 16:50:08.742522 24064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:08.744129 24048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:08.744314 24048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "a065eee9cdf94d3ab166f363ca1d52b0"
format_stamp: "Formatted at 2026-06-19 16:50:06 on dist-test-slave-qb36"
server_key: "b817eb734c93e3ff04b2978ada1daccd"
server_key_iv: "e54b32fbbe8bce7e86c99a7fdbf17721"
server_key_version: "encryptionkey@0"
I20260619 16:50:08.745294 24048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:08.759619 24048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:08.763932 24048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:08.764219 24048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:08.765293 24048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:08.766824 24048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:08.766942 24048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:08.767088 24048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:08.767187 24048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:08.835256 24048 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:41593
I20260619 16:50:08.835346 24176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:41593 every 8 connection(s)
I20260619 16:50:08.838981 24048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:08.847925 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 24048
Jun 19 16:50:08 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:08 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:08.934808 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:08.849882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.14.153.62:36833)
0619 16:50:08.893249 (+ 43367us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:08.893260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:08.893289 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:50:08.893381 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:08.893391 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:08.893525 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:08.893710 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:08.903631 (+ 9921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.905426 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.910763 (+ 5337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.911833 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.912044 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:08.919414 (+ 7370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:08.919432 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:08.919468 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:08.919548 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:08.925941 (+ 6393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:08.928152 (+ 2211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:08.928164 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:08.928175 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:08.928466 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:08.929203 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:08.929211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:08.929217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:08.930030 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:08.930470 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:08.932440 (+ 1970us) server_negotiation.cc:299] Negotiation successful
0619 16:50:08.933166 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43205,"thread_start_us":40987,"threads_started":1}
W20260619 16:50:09.148590 23914 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.333s user 0.004s sys 0.015s
W20260619 16:50:09.150041 23913 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:07.799307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:48733 (local address 127.14.153.62:36833)
0619 16:50:07.800123 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:07.800140 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:07.803942 (+ 3802us) server_negotiation.cc:407] Connection header received
0619 16:50:07.806438 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:07.806452 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:07.806882 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:07.807284 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:07.809321 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:07.810611 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:07.813285 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:07.813997 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:07.814161 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.149731 (+1335570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:48733: BlockingRecv error: recv got EOF from 127.14.153.1:48733 (error 108)
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":305,"threads_started":1}
I20260619 16:50:09.150588 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:07.799442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:48733)
0619 16:50:07.803707 (+ 4265us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:07.803760 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:07.805824 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:07.807496 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:07.807510 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:07.808305 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:07.809081 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:07.809109 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:07.810805 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:07.810813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:07.813084 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:07.813103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:07.814285 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:07.814295 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:07.814632 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:07.815516 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:07.815559 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:09.149404 (+1333845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3952,"thread_start_us":3781,"threads_started":1}
W20260619 16:50:09.151513 23911 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:36833 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.153.62:36833: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:50:09 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:09 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:09.161659 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:09.152251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:40073)
0619 16:50:09.152428 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:09.152448 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:09.152568 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:09.153341 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:09.153347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:09.153816 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:09.154038 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:09.154051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.155409 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.155414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:09.156360 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:09.156371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.157069 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.157082 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:09.157255 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.157868 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:09.157884 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:09.161358 (+ 3474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 16:50:09.161823 23913 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:09.152709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:40073 (local address 127.14.153.62:36833)
0619 16:50:09.152849 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:09.152856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:09.152875 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:50:09.152933 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:09.152938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:09.153009 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:09.153100 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:09.154202 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.155272 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.156504 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.156898 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.157038 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.161551 (+ 4513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:40073: BlockingRecv error: recv got EOF from 127.14.153.1:40073 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 16:50:09 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:09 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:09.171463 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:09.162676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:35615)
0619 16:50:09.162836 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:09.162856 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:09.162947 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:09.163359 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:09.163365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:09.163674 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:09.163816 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:09.163824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.164796 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.164802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:09.166163 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:09.166177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.167214 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.167218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:09.167335 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.167933 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:09.167950 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:09.171169 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 16:50:09.171613 23913 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:09.162925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:35615 (local address 127.14.153.62:36833)
0619 16:50:09.163105 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:09.163112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:09.163131 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:50:09.163170 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:09.163173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:09.163224 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:09.163328 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:09.163951 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.164659 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.166318 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.166928 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.166998 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.171356 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:35615: BlockingRecv error: recv got EOF from 127.14.153.1:35615 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 16:50:09.172259 23911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:50:09.763829 24046 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.424s user 0.010s sys 0.009s
W20260619 16:50:09.765620 24047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:08.313170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:47035 (local address 127.14.153.62:36833)
0619 16:50:08.328135 (+ 14965us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:08.328143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:08.328168 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:50:08.328238 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:08.328245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:08.328350 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:08.328478 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:08.331722 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.333639 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.336511 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.337526 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.337727 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.765336 (+1427609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:47035: BlockingRecv error: recv got EOF from 127.14.153.2:47035 (error 108)
Metrics: {"server-negotiator.queue_time_us":14789,"thread_start_us":15450,"threads_started":1}
I20260619 16:50:09.766803 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:08.312547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:47035)
0619 16:50:08.319975 (+ 7428us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:08.320074 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:08.323663 (+ 3589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:08.329653 (+ 5990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:08.329669 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:08.330519 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:08.331434 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:08.331463 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.333892 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.333901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:08.336309 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:08.336330 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.337890 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.337978 (+ 88us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:08.338408 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:08.339431 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:08.339476 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:09.765166 (+1425690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6951,"thread_start_us":6634,"threads_started":1}
W20260619 16:50:09.767824 24044 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:36833 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.153.62:36833: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:50:09 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:09 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:09.784190 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:09.768457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:57235)
0619 16:50:09.768742 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:09.768771 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:09.768946 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:09.769293 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:09.769301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:09.769963 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:09.770277 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:09.770294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.771416 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.771424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:09.773699 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:09.773723 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.774507 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.774517 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:09.774776 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.778497 (+ 3721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:09.778525 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:09.783878 (+ 5353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260619 16:50:09.785252 24047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:09.768613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:57235 (local address 127.14.153.62:36833)
0619 16:50:09.768747 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:09.768751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:09.768842 (+ 91us) server_negotiation.cc:407] Connection header received
0619 16:50:09.769078 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:09.769082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:09.769140 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:09.769217 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:09.770443 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.771259 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.773882 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.774283 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.774384 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.784684 (+ 10300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:57235: BlockingRecv error: recv got EOF from 127.14.153.2:57235 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 16:50:09 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:09 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:09.797072 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:09.785924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:43447)
0619 16:50:09.786094 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:09.786116 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:09.786233 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:09.786895 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:09.786901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:09.787364 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:09.787595 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:09.787622 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.788741 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.788747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:09.791177 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:09.791192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.792526 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.792530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:09.792662 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.793289 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:09.793306 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:09.796786 (+ 3480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 16:50:09.797288 24047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:09.786220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:43447 (local address 127.14.153.62:36833)
0619 16:50:09.786451 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:09.786460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:09.786486 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:50:09.786572 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:09.786580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:09.786685 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:09.786836 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:09.787765 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.788570 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.791320 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:09.792345 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:09.792534 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:09.796990 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:43447: BlockingRecv error: recv got EOF from 127.14.153.2:43447 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 16:50:09.797683 24044 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 16:50:10 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:10 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:10.183476 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:10.172904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:47063)
0619 16:50:10.173605 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:10.173628 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:10.173750 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:10.174178 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:10.174182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:10.174566 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:10.174714 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:10.174722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.176191 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.176196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:10.177676 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:10.177694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.178724 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.178730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:10.178852 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:10.179608 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:10.179628 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:10.183233 (+ 3605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":417,"threads_started":1}
W20260619 16:50:10.183774 24047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:10.173090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:47063 (local address 127.14.153.62:36833)
0619 16:50:10.173319 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:10.173326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:10.173728 (+ 402us) server_negotiation.cc:407] Connection header received
0619 16:50:10.173866 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:10.173872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:10.173963 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:10.174081 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:10.174865 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.176030 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.177841 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.178485 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.178619 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:10.183521 (+ 4902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:47063: BlockingRecv error: recv got EOF from 127.14.153.1:47063 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 16:50:10.185820 24179 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.260s user 0.006s sys 0.017s
W20260619 16:50:10.186915 24182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:08.845263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:32865 (local address 127.14.153.62:36833)
0619 16:50:08.890845 (+ 45582us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:08.890856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:08.902918 (+ 12062us) server_negotiation.cc:407] Connection header received
0619 16:50:08.903012 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:08.903020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:08.903157 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:08.903301 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:08.914185 (+ 10884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.916092 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.921832 (+ 5740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.923344 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.923509 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:10.186661 (+1263152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:32865: BlockingRecv error: recv got EOF from 127.14.153.3:32865 (error 108)
Metrics: {"server-negotiator.queue_time_us":45395,"thread_start_us":48705,"threads_started":1}
I20260619 16:50:10.187450 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:08.844343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:32865)
0619 16:50:08.898158 (+ 53815us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:08.898234 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:08.902198 (+ 3964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:08.911260 (+ 9062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:08.911282 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:08.912558 (+ 1276us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:08.913883 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:08.913924 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.916402 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.916412 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:08.921577 (+ 5165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:08.921604 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:08.924112 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:08.924123 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:08.924628 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:08.925678 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:08.925725 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:10.186457 (+1260732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53274,"thread_start_us":61078,"threads_started":1}
W20260619 16:50:10.188323 24177 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:36833 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.153.62:36833: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:50:10 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:10 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:10.197606 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:10.188983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:60363)
0619 16:50:10.189158 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:10.189178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:10.189289 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:10.189655 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:10.189660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:10.190126 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:10.190338 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:10.190353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.191593 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.191597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:10.192572 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:10.192583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.193245 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.193250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:10.193369 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:10.193973 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:10.193992 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:10.197383 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 16:50:10.197875 24182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:10.189071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:60363 (local address 127.14.153.62:36833)
0619 16:50:10.189300 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:10.189305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:10.189325 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:50:10.189384 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:10.189388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:10.189464 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:10.189560 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:10.190512 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.191458 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.192695 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.193069 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.193167 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:10.197636 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:60363: BlockingRecv error: recv got EOF from 127.14.153.3:60363 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Jun 19 16:50:10 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:10 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:10.207643 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:10.198269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:36905)
0619 16:50:10.198850 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:10.198866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:10.198963 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:10.199291 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:10.199295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:10.199695 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:10.199922 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:10.199930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.200919 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.200923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:10.202264 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:10.202296 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.203317 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.203321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:10.203442 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:10.203973 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:10.203989 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:10.207398 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":261,"threads_started":1}
W20260619 16:50:10.207844 24182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:10.198359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:36905 (local address 127.14.153.62:36833)
0619 16:50:10.198508 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:10.198512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:10.198944 (+ 432us) server_negotiation.cc:407] Connection header received
0619 16:50:10.199116 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:10.199119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:10.199176 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:10.199255 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:10.200061 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.200797 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.202404 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.203165 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.203259 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:10.207586 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:36905: BlockingRecv error: recv got EOF from 127.14.153.3:36905 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 16:50:10.208144 24177 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 16:50:10 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:10 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:10.807420 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:10.798240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:33643)
0619 16:50:10.798870 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:10.798893 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:10.799055 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:10.799410 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:10.799415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:10.799877 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:10.800017 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:10.800025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.801027 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.801035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:10.802069 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:10.802081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.802806 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.802813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:10.803130 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:10.803789 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:10.803807 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:10.807189 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":340,"threads_started":1}
W20260619 16:50:10.807650 24187 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:10.798346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:33643 (local address 127.14.153.62:36833)
0619 16:50:10.798935 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:10.798943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:10.798963 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:50:10.799193 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:10.799198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:10.799269 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:10.799384 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:10.800161 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.800877 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.802212 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:10.802621 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:10.802767 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:10.807403 (+ 4636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:33643: BlockingRecv error: recv got EOF from 127.14.153.2:33643 (error 108)
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":372,"threads_started":1}
Jun 19 16:50:11 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:11 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:11.193318 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:11.184421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:44865)
0619 16:50:11.184950 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:11.184974 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:11.185082 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:11.185451 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:11.185455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:11.185858 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:11.186063 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:11.186073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.187178 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.187183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:11.188162 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:11.188175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.188819 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.188823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:11.188944 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:11.189546 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:11.189565 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:11.192953 (+ 3388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":327,"threads_started":1}
W20260619 16:50:11.193399 24187 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:11.184502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:44865 (local address 127.14.153.62:36833)
0619 16:50:11.184707 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:11.184713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:11.185038 (+ 325us) server_negotiation.cc:407] Connection header received
0619 16:50:11.185202 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:11.185208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:11.185279 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:11.185366 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:11.186251 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.186986 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.188285 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.188675 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.188764 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:11.193165 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:44865: BlockingRecv error: recv got EOF from 127.14.153.1:44865 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:50:11 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:11 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:11.217698 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:11.208856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:44793)
0619 16:50:11.209464 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:11.209480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:11.209635 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:11.209988 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:11.209993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:11.210367 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:11.210516 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:11.210524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.211626 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.211632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:11.212563 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:11.212574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.213274 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.213280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:11.213444 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:11.214085 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:11.214102 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:11.217482 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":377,"threads_started":1}
W20260619 16:50:11.217919 24187 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:11.208895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:44793 (local address 127.14.153.62:36833)
0619 16:50:11.209047 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:11.209051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:11.209567 (+ 516us) server_negotiation.cc:407] Connection header received
0619 16:50:11.209778 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:11.209783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:11.209856 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:11.209959 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:11.210651 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.211491 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.212698 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.213121 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.213256 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:11.217669 (+ 4413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:44793: BlockingRecv error: recv got EOF from 127.14.153.3:44793 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 16:50:11 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:11 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:11.817600 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:11.808450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:44305)
0619 16:50:11.808951 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:11.809002 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:11.809134 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:11.809666 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:11.809670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:11.810176 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:11.810406 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:11.810418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.811813 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.811818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:11.812738 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:11.812750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.813421 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.813425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:11.813547 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:11.814127 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:11.814144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:11.817336 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":256,"threads_started":1}
W20260619 16:50:11.817935 24191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:11.808534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:44305 (local address 127.14.153.62:36833)
0619 16:50:11.809264 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:11.809268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:11.809307 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:50:11.809371 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:11.809376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:11.809453 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:11.809606 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:11.810580 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.811686 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.812881 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:11.813275 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:11.813369 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:11.817560 (+ 4191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:44305: BlockingRecv error: recv got EOF from 127.14.153.2:44305 (error 108)
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":544,"threads_started":1}
Jun 19 16:50:12 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:12 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:12.203318 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:12.194367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:36569)
0619 16:50:12.194886 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:12.194903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:12.195021 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:12.195335 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:12.195338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:12.195696 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:12.195859 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:12.195866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.196837 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.196841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:12.197779 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:12.197790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.198444 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.198448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:12.198583 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:12.199255 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:12.199271 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:12.203097 (+ 3826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":289,"threads_started":1}
W20260619 16:50:12.203464 24191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:12.194502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:36569 (local address 127.14.153.62:36833)
0619 16:50:12.194754 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:12.194760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:12.194986 (+ 226us) server_negotiation.cc:407] Connection header received
0619 16:50:12.195150 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:12.195154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:12.195213 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:12.195294 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:12.195975 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.196716 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.197923 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.198314 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.198412 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:12.203265 (+ 4853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:36569: BlockingRecv error: recv got EOF from 127.14.153.1:36569 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 19 16:50:12 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:12 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:12.227701 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:12.218659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:45139)
0619 16:50:12.219187 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:12.219203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:12.219352 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:12.219715 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:12.219719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:12.220132 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:12.220296 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:12.220306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.221338 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.221343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:12.222331 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:12.222342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.223154 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.223160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:12.223339 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:12.223915 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:12.223932 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:12.227452 (+ 3520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":272,"threads_started":1}
W20260619 16:50:12.228012 24191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:12.218713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:45139 (local address 127.14.153.62:36833)
0619 16:50:12.218942 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:12.218951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:12.219281 (+ 330us) server_negotiation.cc:407] Connection header received
0619 16:50:12.219477 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:12.219488 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:12.219559 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:12.219690 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:12.220454 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.221201 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.222475 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.222958 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.223115 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:12.227637 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:45139: BlockingRecv error: recv got EOF from 127.14.153.3:45139 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 16:50:12 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:12 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:12.827843 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:12.818584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:44863)
0619 16:50:12.819097 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:12.819120 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:12.819259 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:12.819696 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:12.819701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:12.820070 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:12.820234 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:12.820242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.821260 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.821265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:12.822242 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:12.822253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.822932 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.822937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:12.823169 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:12.823850 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:12.823868 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:12.827482 (+ 3614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
W20260619 16:50:12.827957 24195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:12.818728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:44863 (local address 127.14.153.62:36833)
0619 16:50:12.819418 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:12.819422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:12.819435 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:50:12.819478 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:12.819481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:12.819550 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:12.819668 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:12.820389 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.821121 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.822374 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:12.822791 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:12.822922 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:12.827694 (+ 4772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:44863: BlockingRecv error: recv got EOF from 127.14.153.2:44863 (error 108)
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":466,"threads_started":1}
Jun 19 16:50:13 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:13 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:13.213330 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:13.204544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:41101)
0619 16:50:13.205089 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:13.205104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:13.205206 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:13.205472 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:13.205475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:13.205839 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:13.205991 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:13.206015 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.207136 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.207140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:13.208140 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:13.208151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.208858 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.208863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:13.209010 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:13.209667 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:13.209685 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:13.213074 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
W20260619 16:50:13.213527 24195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:13.204627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:41101 (local address 127.14.153.62:36833)
0619 16:50:13.204847 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:13.204853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:13.205161 (+ 308us) server_negotiation.cc:407] Connection header received
0619 16:50:13.205296 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:13.205300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:13.205358 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:13.205427 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:13.206171 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.206940 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.208276 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.208698 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.208798 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:13.213304 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:41101: BlockingRecv error: recv got EOF from 127.14.153.1:41101 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 16:50:13 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:13 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:13.237757 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:13.228698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:54801)
0619 16:50:13.229200 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:13.229221 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:13.229355 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:13.229738 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:13.229742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:13.230138 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:13.230298 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:13.230307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.231389 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.231395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:13.232397 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:13.232415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.233083 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.233089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:13.233282 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:13.233900 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:13.233918 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:13.237394 (+ 3476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
W20260619 16:50:13.238061 24195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:13.228734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:54801 (local address 127.14.153.62:36833)
0619 16:50:13.228932 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:13.228938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:13.229289 (+ 351us) server_negotiation.cc:407] Connection header received
0619 16:50:13.229530 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:13.229534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:13.229604 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:13.229708 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:13.230449 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.231247 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.232531 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.232941 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.233071 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:13.237607 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:54801: BlockingRecv error: recv got EOF from 127.14.153.3:54801 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 16:50:13 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:13 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:13.838759 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:13.828929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:37407)
0619 16:50:13.829366 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:13.829383 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:13.829479 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:13.829967 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:13.829970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:13.830408 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:13.830647 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:13.830668 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.832135 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.832168 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:13.833377 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:13.833388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.834204 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.834209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:13.834340 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:13.834986 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:13.835002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:13.838500 (+ 3498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":243,"threads_started":1}
W20260619 16:50:13.839068 24199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:13.829009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:37407 (local address 127.14.153.62:36833)
0619 16:50:13.829694 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:13.829698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:13.829714 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:13.829760 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:13.829764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:13.829829 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:13.829927 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:13.830823 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.831999 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.833551 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:13.834058 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:13.834164 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:13.838760 (+ 4596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:37407: BlockingRecv error: recv got EOF from 127.14.153.2:37407 (error 108)
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":502,"threads_started":1}
Jun 19 16:50:14 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:14 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:14.223340 24200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:14.214367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:53421)
0619 16:50:14.214897 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:14.214913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:14.215056 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:14.215378 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:14.215382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:14.215819 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:14.215988 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:14.215996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.216949 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.216953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:14.217911 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:14.217930 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.218629 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.218635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:14.218791 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:14.219407 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:14.219426 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:14.223083 (+ 3657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":285,"threads_started":1}
W20260619 16:50:14.223414 24199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:14.214440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:53421 (local address 127.14.153.62:36833)
0619 16:50:14.214601 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:14.214606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:14.214990 (+ 384us) server_negotiation.cc:407] Connection header received
0619 16:50:14.215152 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:14.215156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:14.215216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:14.215296 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:14.216114 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.216825 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.218067 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.218451 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.218546 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:14.223242 (+ 4696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:53421: BlockingRecv error: recv got EOF from 127.14.153.1:53421 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 16:50:14 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:14 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:14.247017 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:14.238713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:35025)
0619 16:50:14.239230 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:14.239246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:14.239354 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:14.239624 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:14.239627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:14.239994 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:14.240149 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:14.240158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.241117 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.241121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:14.242132 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:14.242144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.242812 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.242816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:14.242933 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:14.243544 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:14.243560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:14.246771 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":253,"threads_started":1}
W20260619 16:50:14.247263 24199 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:14.238785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:35025 (local address 127.14.153.62:36833)
0619 16:50:14.239005 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:14.239012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:14.239325 (+ 313us) server_negotiation.cc:407] Connection header received
0619 16:50:14.239451 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:14.239455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:14.239529 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:14.239611 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:14.240277 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.240983 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.242269 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.242674 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.242765 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:14.246946 (+ 4181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:35025: BlockingRecv error: recv got EOF from 127.14.153.3:35025 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 16:50:14 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:14 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:14.852914 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:14.839854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:53765)
0619 16:50:14.840432 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:14.840455 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:14.840598 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:14.841115 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:14.841126 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:14.841690 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:14.841888 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:14.841901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.842984 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.842989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:14.843951 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:14.843964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.844628 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.844633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:14.844756 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:14.848360 (+ 3604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:14.848405 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:14.852596 (+ 4191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":274,"threads_started":1}
W20260619 16:50:14.853235 24203 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:14.840086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:53765 (local address 127.14.153.62:36833)
0619 16:50:14.840637 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:14.840643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:14.840659 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:14.840709 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:14.840716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:14.840806 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:14.840899 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:14.842075 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.842849 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.844094 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.844505 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.844601 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:14.852870 (+ 8269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:53765: BlockingRecv error: recv got EOF from 127.14.153.2:53765 (error 108)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":346,"threads_started":1}
Jun 19 16:50:15 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:15 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:15.232803 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.224321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:44997)
0619 16:50:15.224765 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:15.224781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:15.224890 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:15.225191 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:15.225195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:15.225572 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:15.225726 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.225733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.226767 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.226771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.227837 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.227849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.228497 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.228501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.228626 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.229206 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:15.229223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:15.232580 (+ 3357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
W20260619 16:50:15.233028 24203 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:15.224518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:44997 (local address 127.14.153.62:36833)
0619 16:50:15.224726 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.224734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.224854 (+ 120us) server_negotiation.cc:407] Connection header received
0619 16:50:15.225004 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.225008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.225068 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.225147 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:15.225880 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.226619 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.227966 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.228361 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.228453 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.232778 (+ 4325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:44997: BlockingRecv error: recv got EOF from 127.14.153.1:44997 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 16:50:15 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:15 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:15.257134 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.248242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:59185)
0619 16:50:15.248757 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:15.248774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:15.248883 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:15.249180 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:15.249183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:15.249682 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:15.249825 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.249834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.250878 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.250881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.251880 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.251891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.252704 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.252708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.252837 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.253461 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:15.253478 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:15.256898 (+ 3420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":303,"threads_started":1}
W20260619 16:50:15.257491 24203 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:15.248274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:59185 (local address 127.14.153.62:36833)
0619 16:50:15.248554 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.248563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.248854 (+ 291us) server_negotiation.cc:407] Connection header received
0619 16:50:15.248979 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.248983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.249062 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.249142 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:15.249965 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.250729 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.252014 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.252568 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.252662 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.257118 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:59185: BlockingRecv error: recv got EOF from 127.14.153.3:59185 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 16:50:15 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:15 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:15.862735 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.853999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:34541)
0619 16:50:15.854509 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:15.854528 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:15.854637 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:15.854948 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:15.854951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:15.855394 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:15.855552 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.855560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.856538 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.856543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.857468 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.857478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.858156 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.858160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.858282 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.858868 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:15.858885 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:15.862500 (+ 3615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":289,"threads_started":1}
W20260619 16:50:15.862982 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:15.854040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:34541 (local address 127.14.153.62:36833)
0619 16:50:15.854542 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.854547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.854584 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:50:15.854742 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.854746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.854816 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.854903 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:15.855684 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.856405 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.857592 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.858012 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.858117 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.862708 (+ 4591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:34541: BlockingRecv error: recv got EOF from 127.14.153.2:34541 (error 108)
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":266,"threads_started":1}
Jun 19 16:50:16 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:16 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:16.243203 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.233966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:58571)
0619 16:50:16.234493 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:16.234508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:16.234676 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:16.235012 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:16.235017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:16.235477 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:16.235659 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.235668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.236707 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.236713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.237772 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.237783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.238482 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.238487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.238683 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.239288 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:16.239305 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:16.242846 (+ 3541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":235,"threads_started":1}
W20260619 16:50:16.243353 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:16.234017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:58571 (local address 127.14.153.62:36833)
0619 16:50:16.234248 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:16.234254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:16.234603 (+ 349us) server_negotiation.cc:407] Connection header received
0619 16:50:16.234797 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:16.234803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:16.234877 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:16.234978 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:16.235805 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.236559 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.237902 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.238329 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.238467 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.243104 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:58571: BlockingRecv error: recv got EOF from 127.14.153.1:58571 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 19 16:50:16 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:16 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:16.266907 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.258112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:38203)
0619 16:50:16.258653 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:16.258669 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:16.258828 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:16.259252 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:16.259256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:16.259674 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:16.259831 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.259840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.260823 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.260828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.261744 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.261755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.262463 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.262469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.262639 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.263300 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:16.263317 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:16.266567 (+ 3250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":323,"threads_started":1}
W20260619 16:50:16.267062 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:16.258192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:38203 (local address 127.14.153.62:36833)
0619 16:50:16.258469 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:16.258477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:16.258754 (+ 277us) server_negotiation.cc:407] Connection header received
0619 16:50:16.258974 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:16.258979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:16.259118 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:16.259222 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:16.259975 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.260687 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.261888 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.262300 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.262424 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.266767 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:38203: BlockingRecv error: recv got EOF from 127.14.153.3:38203 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 19 16:50:16 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:16 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:16.873298 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.863701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:40577)
0619 16:50:16.864304 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:16.864322 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:16.864445 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:16.864952 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:16.864956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:16.865396 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:16.865602 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.865611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.866586 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.866589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.867617 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.867629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.868286 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.868290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.868440 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.869127 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:16.869144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:16.872949 (+ 3805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":238,"threads_started":1}
W20260619 16:50:16.873417 24210 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:16.863779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:40577 (local address 127.14.153.62:36833)
0619 16:50:16.864625 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:16.864629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:16.864645 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:16.864691 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:16.864694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:16.864767 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:16.864872 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:16.865735 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.866445 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.867738 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.868144 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.868244 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.873137 (+ 4893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:40577: BlockingRecv error: recv got EOF from 127.14.153.2:40577 (error 108)
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":614,"threads_started":1}
Jun 19 16:50:17 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:17 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:17.253353 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.244326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:57163)
0619 16:50:17.244864 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.244889 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:17.245001 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:17.245289 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:17.245292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:17.245672 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:17.245831 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.245839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.246833 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.246836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.247776 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.247788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.248483 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.248495 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.248640 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.249229 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:17.249247 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:17.252996 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":287,"threads_started":1}
W20260619 16:50:17.253445 24210 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:17.244421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:57163 (local address 127.14.153.62:36833)
0619 16:50:17.244591 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.244596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.244961 (+ 365us) server_negotiation.cc:407] Connection header received
0619 16:50:17.245103 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.245107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.245165 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.245247 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:17.245963 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.246711 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.247894 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.248294 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.248383 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.253186 (+ 4803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:57163: BlockingRecv error: recv got EOF from 127.14.153.1:57163 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 16:50:17 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:17 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:17.276839 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.267879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:47571)
0619 16:50:17.268510 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.268526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:17.268647 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:17.268943 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:17.268946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:17.269338 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:17.269498 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.269506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.270429 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.270432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.271427 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.271439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.272043 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.272047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.272156 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.272702 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:17.272719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:17.276489 (+ 3770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":311,"threads_started":1}
W20260619 16:50:17.277045 24210 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:17.268048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:47571 (local address 127.14.153.62:36833)
0619 16:50:17.268204 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.268208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.268600 (+ 392us) server_negotiation.cc:407] Connection header received
0619 16:50:17.268748 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.268751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.268819 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.268897 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:17.269611 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.270310 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.271545 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.271926 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.272014 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.276755 (+ 4741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:47571: BlockingRecv error: recv got EOF from 127.14.153.3:47571 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 16:50:17 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:17 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:17.884045 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.874513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:60571)
0619 16:50:17.875135 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.875157 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:17.875314 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:17.875804 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:17.875809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:17.876351 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:17.876545 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.876555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.877719 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.877723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.878634 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.878645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.879652 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.879657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.879770 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.880437 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:17.880455 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:17.883801 (+ 3346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":284,"threads_started":1}
W20260619 16:50:17.884342 24215 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:17.874723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:60571 (local address 127.14.153.62:36833)
0619 16:50:17.875246 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.875253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.875274 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:50:17.875483 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.875487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.875610 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.875734 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:17.876673 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.877593 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.878799 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.879497 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.879592 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.883998 (+ 4406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:60571: BlockingRecv error: recv got EOF from 127.14.153.2:60571 (error 108)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":232,"threads_started":1}
Jun 19 16:50:18 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:18 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:18.263803 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:18.254438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:52411)
0619 16:50:18.254915 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:18.254931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:18.255108 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:18.255418 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:18.255422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:18.255831 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:18.255971 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:18.255979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.257201 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.257207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:18.258366 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:18.258380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.259310 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.259323 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:18.259450 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:18.260027 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:18.260044 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:18.263570 (+ 3526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":275,"threads_started":1}
W20260619 16:50:18.264029 24215 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:18.254467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:52411 (local address 127.14.153.62:36833)
0619 16:50:18.254705 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:18.254715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:18.255017 (+ 302us) server_negotiation.cc:407] Connection header received
0619 16:50:18.255225 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:18.255229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:18.255292 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:18.255386 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:18.256118 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.257011 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.258507 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.259075 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.259231 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:18.263796 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:52411: BlockingRecv error: recv got EOF from 127.14.153.1:52411 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 16:50:18 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:18 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:18.287562 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:18.278144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:44831)
0619 16:50:18.278631 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:18.278647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:18.278770 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:18.279153 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:18.279156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:18.279513 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:18.279691 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:18.279700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.280826 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.280830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:18.281987 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:18.281999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.282972 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.282980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:18.283170 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:18.283909 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:18.283926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:18.287308 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
W20260619 16:50:18.287873 24215 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:18.278258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:44831 (local address 127.14.153.62:36833)
0619 16:50:18.278523 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:18.278530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:18.278763 (+ 233us) server_negotiation.cc:407] Connection header received
0619 16:50:18.278869 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:18.278873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:18.278955 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:18.279124 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:18.279837 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.280709 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.282151 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.282751 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.282910 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:18.287570 (+ 4660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:44831: BlockingRecv error: recv got EOF from 127.14.153.3:44831 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 16:50:18 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:18 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:18.894769 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:18.885179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:42371)
0619 16:50:18.885987 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:18.886005 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:18.886103 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:18.886486 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:18.886491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:18.886929 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:18.887110 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:18.887119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.888115 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.888120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:18.889052 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:18.889063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.889806 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.889811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:18.889990 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:18.890719 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:18.890738 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:18.894441 (+ 3703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":385,"threads_started":1}
W20260619 16:50:18.894976 24219 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:18.885261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:42371 (local address 127.14.153.62:36833)
0619 16:50:18.886163 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:18.886169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:18.886185 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:18.886230 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:18.886236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:18.886319 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:18.886451 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:18.887258 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.887965 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.889208 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.889648 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.889775 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:18.894687 (+ 4912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:42371: BlockingRecv error: recv got EOF from 127.14.153.2:42371 (error 108)
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":671,"threads_started":1}
Jun 19 16:50:19 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:19 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:19.272689 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.264751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:46851)
0619 16:50:19.265266 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.265282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.265390 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.265654 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.265657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.265975 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:19.266145 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.266153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.267134 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.267138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.268099 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.268109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.268691 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.268695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.268805 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.269340 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:19.269357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:19.272451 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":285,"threads_started":1}
W20260619 16:50:19.272810 24219 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:19.264762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:46851 (local address 127.14.153.62:36833)
0619 16:50:19.264973 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.264982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.265363 (+ 381us) server_negotiation.cc:407] Connection header received
0619 16:50:19.265471 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.265474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.265532 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.265618 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:19.266260 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.266974 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.268208 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.268575 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.268661 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.272604 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:46851: BlockingRecv error: recv got EOF from 127.14.153.1:46851 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 16:50:19 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:19 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:19.297323 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.288574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:59581)
0619 16:50:19.289144 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.289165 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.289321 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.289664 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.289668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.290119 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:19.290271 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.290280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.291331 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.291338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.292280 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.292292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.292993 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.292999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.293180 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.293781 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:19.293797 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:19.297089 (+ 3292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":309,"threads_started":1}
W20260619 16:50:19.297627 24219 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:19.288698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:59581 (local address 127.14.153.62:36833)
0619 16:50:19.288940 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.288947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.289262 (+ 315us) server_negotiation.cc:407] Connection header received
0619 16:50:19.289451 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.289456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.289536 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.289640 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:19.290417 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.291209 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.292419 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.292838 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.292961 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.297320 (+ 4359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:59581: BlockingRecv error: recv got EOF from 127.14.153.3:59581 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 16:50:19 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:19 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:19.908579 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.895821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:33555)
0619 16:50:19.896369 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.896386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.896494 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.896808 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.896811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.897193 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:19.897338 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.897361 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.898585 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.898589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.899625 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.899637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.900280 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.900284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.900389 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.904676 (+ 4287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:19.904700 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:19.908340 (+ 3640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":299,"threads_started":1}
W20260619 16:50:19.908783 24223 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:19.895873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:33555 (local address 127.14.153.62:36833)
0619 16:50:19.896395 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.896399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.896436 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:50:19.896600 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.896604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.896672 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.896763 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:19.897496 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.898435 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.899757 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.900156 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.900250 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.908558 (+ 8308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:33555: BlockingRecv error: recv got EOF from 127.14.153.2:33555 (error 108)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":296,"threads_started":1}
Jun 19 16:50:20 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:20 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:20.282582 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.273632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:33653)
0619 16:50:20.274161 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.274177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.274329 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.274664 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.274669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.275187 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:20.275359 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.275368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.276347 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.276354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.277273 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.277283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.278005 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.278012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.278186 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.278761 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:20.278778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:20.282230 (+ 3452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":302,"threads_started":1}
W20260619 16:50:20.282675 24223 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:20.273819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:33653 (local address 127.14.153.62:36833)
0619 16:50:20.274030 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.274037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.274255 (+ 218us) server_negotiation.cc:407] Connection header received
0619 16:50:20.274460 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.274465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.274523 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.274640 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:20.275500 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.276213 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.277428 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.277851 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.277975 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.282427 (+ 4452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:33653: BlockingRecv error: recv got EOF from 127.14.153.1:33653 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 16:50:20 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:20 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:20.307400 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.298222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:43149)
0619 16:50:20.298715 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.298731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.298878 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.299240 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.299245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.299626 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:20.299791 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.299800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.300825 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.300832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.301919 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.301931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.302671 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.302676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.302844 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.303511 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:20.303528 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:20.307089 (+ 3561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":255,"threads_started":1}
W20260619 16:50:20.307783 24223 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:20.298274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:43149 (local address 127.14.153.62:36833)
0619 16:50:20.298481 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.298489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.298809 (+ 320us) server_negotiation.cc:407] Connection header received
0619 16:50:20.298990 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.298996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.299107 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.299212 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:20.299934 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.300672 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.302049 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.302522 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.302656 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.307318 (+ 4662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:43149: BlockingRecv error: recv got EOF from 127.14.153.3:43149 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 16:50:20 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:20 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:20.919116 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.909683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:35235)
0619 16:50:20.910187 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.910204 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.910319 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.910912 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.910916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.911458 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:20.911755 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.911767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.912996 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.913000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.913889 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.913900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.914582 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.914587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.914703 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.915340 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:20.915357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:20.918872 (+ 3515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":281,"threads_started":1}
W20260619 16:50:20.919414 24227 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:20.909822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:35235 (local address 127.14.153.62:36833)
0619 16:50:20.910570 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.910575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.910593 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:50:20.910645 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.910649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.910706 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.910865 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:20.911937 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.912870 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.914022 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.914439 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.914535 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.919113 (+ 4578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:35235: BlockingRecv error: recv got EOF from 127.14.153.2:35235 (error 108)
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":517,"threads_started":1}
Jun 19 16:50:21 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:21 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:21.292249 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:21.283600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:33113)
0619 16:50:21.284069 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:21.284085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:21.284195 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:21.284487 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:21.284490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:21.284910 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:21.285073 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:21.285081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.286024 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.286028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:21.286932 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:21.286942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.287662 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.287667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:21.287803 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:21.288390 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:21.288406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:21.291995 (+ 3589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
W20260619 16:50:21.292487 24227 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:21.283655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:33113 (local address 127.14.153.62:36833)
0619 16:50:21.283887 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:21.283893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:21.284164 (+ 271us) server_negotiation.cc:407] Connection header received
0619 16:50:21.284296 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:21.284300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:21.284362 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:21.284444 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:21.285201 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.285907 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.287113 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.287494 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.287587 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:21.292230 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:33113: BlockingRecv error: recv got EOF from 127.14.153.1:33113 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:50:21 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:21 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:21.317806 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:21.309053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:50767)
0619 16:50:21.309545 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:21.309562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:21.309716 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:21.310066 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:21.310071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:21.310476 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:21.310645 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:21.310654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.311729 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.311737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:21.312641 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:21.312651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.313381 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.313386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:21.313582 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:21.314236 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:21.314253 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:21.317589 (+ 3336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":297,"threads_started":1}
W20260619 16:50:21.318009 24227 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:21.309158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:50767 (local address 127.14.153.62:36833)
0619 16:50:21.309363 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:21.309372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:21.309652 (+ 280us) server_negotiation.cc:407] Connection header received
0619 16:50:21.309846 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:21.309850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:21.309926 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:21.310035 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:21.310786 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.311600 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.312785 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.313230 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.313367 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:21.317763 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:50767: BlockingRecv error: recv got EOF from 127.14.153.3:50767 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 16:50:21 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:21 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:21.929194 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:21.920022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:39041)
0619 16:50:21.920535 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:21.920569 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:21.920672 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:21.921151 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:21.921159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:21.921574 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:21.921737 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:21.921745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.923299 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.923304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:21.924351 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:21.924363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.924998 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.925002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:21.925147 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:21.925705 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:21.925722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:21.928950 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":296,"threads_started":1}
W20260619 16:50:21.929473 24231 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:21.920237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:39041 (local address 127.14.153.62:36833)
0619 16:50:21.920860 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:21.920864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:21.920881 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:50:21.920931 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:21.920934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:21.920990 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:21.921082 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:21.921875 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.923157 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.924471 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:21.924881 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:21.924973 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:21.929177 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:39041: BlockingRecv error: recv got EOF from 127.14.153.2:39041 (error 108)
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":429,"threads_started":1}
Jun 19 16:50:22 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:22 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:22.302168 24232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:22.293134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:32807)
0619 16:50:22.293641 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:22.293658 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:22.293770 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:22.294084 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:22.294087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:22.294448 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:22.294629 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.294637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.295766 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.295770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.296671 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.296682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.297382 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.297387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.297506 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.298147 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:22.298164 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:22.301870 (+ 3706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":268,"threads_started":1}
W20260619 16:50:22.302366 24231 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:22.293248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:32807 (local address 127.14.153.62:36833)
0619 16:50:22.293427 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:22.293431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:22.293767 (+ 336us) server_negotiation.cc:407] Connection header received
0619 16:50:22.293862 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:22.293866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:22.293956 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:22.294043 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:22.294784 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.295604 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.296804 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.297194 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.297287 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.302131 (+ 4844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:32807: BlockingRecv error: recv got EOF from 127.14.153.1:32807 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 16:50:22 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:22 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:22.327564 24233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:22.318755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:45645)
0619 16:50:22.319260 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:22.319277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:22.319388 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:22.319678 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:22.319681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:22.320073 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:22.320246 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.320254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.321304 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.321308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.322229 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.322240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.322944 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.322948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.323136 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.323708 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:22.323725 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:22.327297 (+ 3572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":299,"threads_started":1}
W20260619 16:50:22.327739 24231 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:22.318803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:45645 (local address 127.14.153.62:36833)
0619 16:50:22.319049 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:22.319058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:22.319351 (+ 293us) server_negotiation.cc:407] Connection header received
0619 16:50:22.319494 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:22.319497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:22.319557 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:22.319636 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:22.320374 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.321097 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.322393 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.322787 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.322883 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.327514 (+ 4631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:45645: BlockingRecv error: recv got EOF from 127.14.153.3:45645 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 16:50:22 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:22 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.2@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:22.939391 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:22.930193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.2:60493)
0619 16:50:22.930749 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:22.930770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:22.930888 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:22.931199 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:22.931202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:22.931585 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:22.931722 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.931730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.932700 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.932704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.933758 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.933771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.934671 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.934678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.934818 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.935596 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:22.935614 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:22.939153 (+ 3539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":253,"threads_started":1}
W20260619 16:50:22.939607 24235 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:22.930422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:60493 (local address 127.14.153.62:36833)
0619 16:50:22.930911 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:22.930916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:22.930933 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:50:22.930981 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:22.930984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:22.931066 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:22.931156 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:22.931870 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.932578 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.933940 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.934516 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.934628 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.939346 (+ 4718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.2:60493: BlockingRecv error: recv got EOF from 127.14.153.2:60493 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":254,"threads_started":1}
Jun 19 16:50:23 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:23 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.1@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:23.316164 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.305547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.1:46777)
0619 16:50:23.306078 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.306098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:23.306225 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:23.306547 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:23.306552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:23.306997 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:23.307248 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.307260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.308351 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.308357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.309867 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.309882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.310602 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.310608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.310796 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.311568 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:23.311587 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:23.315815 (+ 4228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
W20260619 16:50:23.316251 24235 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:23.305614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:46777 (local address 127.14.153.62:36833)
0619 16:50:23.305855 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.305863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.306204 (+ 341us) server_negotiation.cc:407] Connection header received
0619 16:50:23.306333 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.306337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.306407 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.306492 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:23.307392 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.308209 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.310008 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.310444 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.310538 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.316011 (+ 5473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.1:46777: BlockingRecv error: recv got EOF from 127.14.153.1:46777 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:50:23 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:50:23 dist-test-slave-qb36 krb5kdc[23176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.153.3@KRBTEST.COM for oryx/127.14.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:50:23.337971 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.328609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:36833 (local address 127.14.153.3:33351)
0619 16:50:23.329090 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.329106 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:23.329217 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:23.329580 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:23.329584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:23.329985 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:23.330138 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.330147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.331263 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.331268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.332221 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.332235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.332916 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.332920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.333057 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.333776 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:23.333795 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:23.337615 (+ 3820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.153.62:36833: Server oryx/127.14.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":273,"threads_started":1}
W20260619 16:50:23.338289 24235 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:23.328660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:33351 (local address 127.14.153.62:36833)
0619 16:50:23.328899 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.328903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.329178 (+ 275us) server_negotiation.cc:407] Connection header received
0619 16:50:23.329329 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.329333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.329404 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.329492 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:23.330273 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.331133 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.332347 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.332774 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.332872 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.338028 (+ 5156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.153.3:33351: BlockingRecv error: recv got EOF from 127.14.153.3:33351 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20260619 16:50:23.858999 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 23764
I20260619 16:50:23.871713 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.860150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.14.153.1:33023)
0619 16:50:23.860858 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.860866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.860894 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:50:23.860960 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.860966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.861202 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.861400 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:23.862756 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.863906 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.865085 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.865632 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.865733 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.868410 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:23.868420 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:23.868433 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:23.868496 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:23.871213 (+ 2717us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.153.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 16:50:23.871503 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44960: Request ticket server kudu/127.14.153.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":403,"threads_started":1}
W20260619 16:50:23.871876 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 23764: Not authorized: Client connection negotiation failed: client connection to 127.14.153.1:33023: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.153.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 16:50:23.872107 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 23764
I20260619 16:50:23.889117 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 23915
I20260619 16:50:23.902022 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.890242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.14.153.2:37731)
0619 16:50:23.890828 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.890835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.890861 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:50:23.890932 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.890939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.891262 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.891471 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:23.892756 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.893770 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.894964 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.895634 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.895782 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.898493 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:23.898503 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:23.898517 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:23.898557 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:23.901304 (+ 2747us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.153.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 16:50:23.901672 (+ 368us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33532: Request ticket server kudu/127.14.153.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":312,"threads_started":1}
W20260619 16:50:23.902052 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 23915: Not authorized: Client connection negotiation failed: client connection to 127.14.153.2:37731: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.153.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 16:50:23.902144 24240 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33532: Request ticket server kudu/127.14.153.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 16:50:23.902187 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 23915
I20260619 16:50:23.918707 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 24048
I20260619 16:50:23.932253 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.919955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.14.153.3:41593)
0619 16:50:23.920622 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.920629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.920656 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:50:23.920721 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.920728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.921001 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.921180 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:23.923232 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.924277 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.925530 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.926121 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.926234 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.928866 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:23.928883 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:23.928896 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:23.928938 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:23.931671 (+ 2733us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.153.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 16:50:23.931956 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49114: Request ticket server kudu/127.14.153.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":345,"threads_started":1}
W20260619 16:50:23.932276 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 24048: Not authorized: Client connection negotiation failed: client connection to 127.14.153.3:41593: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.153.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 16:50:23.932387 24241 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49114: Request ticket server kudu/127.14.153.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 16:50:23.932432 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 24048
I20260619 16:50:23.948478 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 23695
I20260619 16:50:23.949930 23757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:24.062547 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 23695
2026-06-19T16:50:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21310 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T16:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:24Z Disabled control of system clock
I20260619 16:50:24.139436 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:41447
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:41447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:50:24.465567 24246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:24.465853 24246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:24.465930 24246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:24.473174 24246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:24.473294 24246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:24.473340 24246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:24.473389 24246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:24.473436 24246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:24.482682 24246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:41447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:41447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:24.484053 24246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:24.485745 24246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:24.497470 24251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:24.497534 24254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:24.497503 24252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:24.498189 24246 server_base.cc:1034] running on GCE node
I20260619 16:50:24.498958 24246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:24.500499 24246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:24.501731 24246 hybrid_clock.cc:648] HybridClock initialized: now 1781887824501668 us; error 61 us; skew 500 ppm
I20260619 16:50:24.502131 24246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:50:24.504465 24246 webserver.cc:460] Webserver started at http://127.14.153.62:46059/ using document root <none> and password file <none>
I20260619 16:50:24.505185 24246 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:24.505301 24246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:24.505643 24246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:24.508625 24246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "3d6b6748f4d4438ea92163bcf206f696"
format_stamp: "Formatted at 2026-06-19 16:50:24 on dist-test-slave-qb36"
server_key: "ca864ed6b8bb3c1cfafbb732e84bdb75"
server_key_iv: "65afa40d1d62f842bda293b0dc5de91f"
server_key_version: "encryptionkey@0"
I20260619 16:50:24.509407 24246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "3d6b6748f4d4438ea92163bcf206f696"
format_stamp: "Formatted at 2026-06-19 16:50:24 on dist-test-slave-qb36"
server_key: "ca864ed6b8bb3c1cfafbb732e84bdb75"
server_key_iv: "65afa40d1d62f842bda293b0dc5de91f"
server_key_version: "encryptionkey@0"
I20260619 16:50:24.514685 24246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:50:24.518069 24260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:24.519441 24246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:24.519608 24246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "3d6b6748f4d4438ea92163bcf206f696"
format_stamp: "Formatted at 2026-06-19 16:50:24 on dist-test-slave-qb36"
server_key: "ca864ed6b8bb3c1cfafbb732e84bdb75"
server_key_iv: "65afa40d1d62f842bda293b0dc5de91f"
server_key_version: "encryptionkey@0"
I20260619 16:50:24.519827 24246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:24.539405 24246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:24.540266 24246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:24.540575 24246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:24.559546 24246 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:41447
I20260619 16:50:24.559564 24311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:41447 every 8 connection(s)
I20260619 16:50:24.561017 24246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:24.561921 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 24246
I20260619 16:50:24.562139 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:50:24.562748 14948 external_mini_cluster.cc:1428] Setting key e0ac64fc92911636d0d19d18c261f15f
I20260619 16:50:24.565655 24312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:24.578651 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.564902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.14.153.62:41447)
0619 16:50:24.565945 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.565960 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.566019 (+ 59us) server_negotiation.cc:407] Connection header received
0619 16:50:24.568303 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.568331 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.568928 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.569419 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:24.569977 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.572029 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.573318 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.574333 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.574518 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.575685 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:24.575696 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:24.575713 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:24.576494 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:24.576601 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:24.576631 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:24.576754 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.576962 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.577358 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":313,"threads_started":1}
I20260619 16:50:24.579664 24312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696: Bootstrap starting.
I20260619 16:50:24.582628 24312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:24.583691 24312 log.cc:824] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:24.586316 24312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696: No bootstrap required, opened a new log
I20260619 16:50:24.593217 24312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6b6748f4d4438ea92163bcf206f696" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41447 } }
I20260619 16:50:24.593614 24312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:24.593734 24312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6b6748f4d4438ea92163bcf206f696, State: Initialized, Role: FOLLOWER
I20260619 16:50:24.594352 24312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6b6748f4d4438ea92163bcf206f696" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41447 } }
I20260619 16:50:24.594585 24312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:24.594708 24312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:24.594882 24312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:24.596977 24312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6b6748f4d4438ea92163bcf206f696" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41447 } }
I20260619 16:50:24.597494 24312 leader_election.cc:302] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d6b6748f4d4438ea92163bcf206f696; no voters:
I20260619 16:50:24.598074 24312 leader_election.cc:288] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:24.598242 24317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:24.598938 24317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [term 1 LEADER]: Becoming Leader. State: Replica: 3d6b6748f4d4438ea92163bcf206f696, State: Running, Role: LEADER
I20260619 16:50:24.599529 24317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6b6748f4d4438ea92163bcf206f696" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41447 } }
I20260619 16:50:24.600116 24312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:24.602399 24318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d6b6748f4d4438ea92163bcf206f696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6b6748f4d4438ea92163bcf206f696" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41447 } } }
I20260619 16:50:24.602783 24318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:24.602505 24319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d6b6748f4d4438ea92163bcf206f696. Latest consensus state: current_term: 1 leader_uuid: "3d6b6748f4d4438ea92163bcf206f696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6b6748f4d4438ea92163bcf206f696" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41447 } } }
I20260619 16:50:24.602993 24319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:24.606508 24322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:24.612035 24322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:24.625065 24322 catalog_manager.cc:1347] Generated new cluster ID: acafc433e9f0469cbe8b32a837e08746
I20260619 16:50:24.625190 24322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:24.657172 24322 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:24.659196 24322 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:24.668823 24322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d6b6748f4d4438ea92163bcf206f696: Generated new TSK 0
I20260619 16:50:24.669919 24322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:24.680804 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41447
--builtin_ntp_servers=127.14.153.20:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 16:50:25.037761 24336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:25.038106 24336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:25.038264 24336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:25.045812 24336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:25.045971 24336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:25.046144 24336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:50:25.055500 24336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:25.056866 24336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:25.058760 24336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:25.072302 24342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:25.072355 24344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:25.072391 24341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:25.073593 24336 server_base.cc:1034] running on GCE node
I20260619 16:50:25.074280 24336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:25.075376 24336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:25.076602 24336 hybrid_clock.cc:648] HybridClock initialized: now 1781887825076558 us; error 51 us; skew 500 ppm
I20260619 16:50:25.076983 24336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:50:25.084343 24336 webserver.cc:460] Webserver started at http://127.14.153.1:44961/ using document root <none> and password file <none>
I20260619 16:50:25.085202 24336 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:25.085350 24336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:25.085704 24336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:25.089134 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "53aaeca1d4fb4fb9a567109df8b7f638"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-qb36"
server_key: "8a64c3b1dc4e2297946adbdd81c23a3f"
server_key_iv: "25fb40c65c6ad64c190fbcaf6f0f421e"
server_key_version: "encryptionkey@0"
I20260619 16:50:25.090023 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "53aaeca1d4fb4fb9a567109df8b7f638"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-qb36"
server_key: "8a64c3b1dc4e2297946adbdd81c23a3f"
server_key_iv: "25fb40c65c6ad64c190fbcaf6f0f421e"
server_key_version: "encryptionkey@0"
I20260619 16:50:25.095526 24336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:50:25.098910 24350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:25.100486 24336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:50:25.100674 24336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "53aaeca1d4fb4fb9a567109df8b7f638"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-qb36"
server_key: "8a64c3b1dc4e2297946adbdd81c23a3f"
server_key_iv: "25fb40c65c6ad64c190fbcaf6f0f421e"
server_key_version: "encryptionkey@0"
I20260619 16:50:25.100896 24336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:25.124054 24336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:25.125012 24336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:25.125327 24336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:25.126303 24336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:25.127910 24336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:25.128041 24336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:25.128167 24336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:25.128254 24336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:25.161543 24336 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:44453
I20260619 16:50:25.161588 24462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:44453 every 8 connection(s)
I20260619 16:50:25.162971 24336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:25.169579 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 24336
I20260619 16:50:25.169783 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:25.170585 14948 external_mini_cluster.cc:1428] Setting key a04ee99bf66408bdbe40f1f7abe81015
I20260619 16:50:25.173565 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41447
--builtin_ntp_servers=127.14.153.20:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:50:25.176393 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:25.165474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:39165 (local address 127.14.153.62:41447)
0619 16:50:25.165931 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:25.165937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:25.167179 (+ 1242us) server_negotiation.cc:407] Connection header received
0619 16:50:25.168858 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:25.168863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:25.168976 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:25.169124 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:25.170580 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.171476 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.173322 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.173832 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.173929 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:25.175671 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:25.175676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:25.175679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:25.175696 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:25.175739 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:25.175750 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:25.175820 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:25.176072 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 16:50:25.176146 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260619 16:50:25.177933 24466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:25.166111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41447 (local address 127.14.153.1:39165)
0619 16:50:25.166972 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:25.167012 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:25.168617 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:25.169404 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:25.169417 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:25.169546 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:25.170400 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:25.170429 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.171652 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.171661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:25.173175 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:25.173193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.174051 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.174060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:25.174407 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:25.175281 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:25.175286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:25.175339 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:25.175465 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:25.175848 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:25.175860 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:25.175871 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 16:50:25.176061 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 16:50:25.176501 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":257,"threads_started":1}
I20260619 16:50:25.179554 24463 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41447
I20260619 16:50:25.179857 24463 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:25.180625 24463 heartbeater.cc:511] Master 127.14.153.62:41447 requested a full tablet report, sending...
I20260619 16:50:25.182878 24277 ts_manager.cc:194] Registered new tserver with Master: 53aaeca1d4fb4fb9a567109df8b7f638 (127.14.153.1:44453)
I20260619 16:50:25.185047 24277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.1:39165
W20260619 16:50:25.506348 24467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:25.506633 24467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:25.506716 24467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:25.514108 24467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:25.514217 24467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:25.514328 24467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:50:25.523702 24467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:25.525012 24467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:25.526770 24467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:25.539714 24472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:25.539695 24473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:25.539928 24475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:25.540506 24467 server_base.cc:1034] running on GCE node
I20260619 16:50:25.541133 24467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:25.542122 24467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:25.543340 24467 hybrid_clock.cc:648] HybridClock initialized: now 1781887825543272 us; error 65 us; skew 500 ppm
I20260619 16:50:25.543678 24467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:50:25.546011 24467 webserver.cc:460] Webserver started at http://127.14.153.2:43093/ using document root <none> and password file <none>
I20260619 16:50:25.546753 24467 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:25.546887 24467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:25.547289 24467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:25.550094 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "91ec118960a549668d0cf9afca27c3fc"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-qb36"
server_key: "3abfd6f1cb7e63f5be5414c5c631c71f"
server_key_iv: "2cf97cb40554302e28739f64d139e889"
server_key_version: "encryptionkey@0"
I20260619 16:50:25.550900 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "91ec118960a549668d0cf9afca27c3fc"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-qb36"
server_key: "3abfd6f1cb7e63f5be5414c5c631c71f"
server_key_iv: "2cf97cb40554302e28739f64d139e889"
server_key_version: "encryptionkey@0"
I20260619 16:50:25.556074 24467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 16:50:25.559700 24481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:25.561292 24467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:50:25.561455 24467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "91ec118960a549668d0cf9afca27c3fc"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-qb36"
server_key: "3abfd6f1cb7e63f5be5414c5c631c71f"
server_key_iv: "2cf97cb40554302e28739f64d139e889"
server_key_version: "encryptionkey@0"
I20260619 16:50:25.561700 24467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:25.579663 24467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:25.580680 24467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:25.580996 24467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:25.582026 24467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:25.583456 24467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:25.583549 24467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:25.583629 24467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:25.583683 24467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:25.616663 24467 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:39081
I20260619 16:50:25.616714 24593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:39081 every 8 connection(s)
I20260619 16:50:25.618041 24467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:25.619834 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 24467
I20260619 16:50:25.620002 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:25.620432 14948 external_mini_cluster.cc:1428] Setting key 1095fcdbe15449df947e3eefec1bed35
I20260619 16:50:25.623376 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41447
--builtin_ntp_servers=127.14.153.20:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:50:25.632386 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:25.621657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:59113 (local address 127.14.153.62:41447)
0619 16:50:25.621852 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:25.621858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:25.623458 (+ 1600us) server_negotiation.cc:407] Connection header received
0619 16:50:25.625250 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:25.625258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:25.625342 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:25.625452 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:25.626467 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.627672 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.629199 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.629757 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.629855 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:25.631496 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:25.631506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:25.631521 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:25.631546 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:25.631589 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:25.631606 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:25.631700 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:25.631898 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 16:50:25.632005 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:50:25.633247 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:25.622159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41447 (local address 127.14.153.2:59113)
0619 16:50:25.623299 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:25.623350 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:25.624993 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:25.625640 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:25.625647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:25.625731 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:25.626299 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:25.626315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.627813 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.627816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:25.629052 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:25.629063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.629982 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.629992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:25.630316 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:25.631178 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:25.631181 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:25.631220 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:25.631298 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:25.631700 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:25.631707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:25.631712 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:50:25.631857 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:50:25.632145 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":338,"threads_started":1}
I20260619 16:50:25.634690 24594 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41447
I20260619 16:50:25.634991 24594 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:25.635777 24594 heartbeater.cc:511] Master 127.14.153.62:41447 requested a full tablet report, sending...
I20260619 16:50:25.637188 24277 ts_manager.cc:194] Registered new tserver with Master: 91ec118960a549668d0cf9afca27c3fc (127.14.153.2:39081)
I20260619 16:50:25.637971 24277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.2:59113
W20260619 16:50:25.955441 24597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:25.955819 24597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:25.955994 24597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:25.963260 24597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:25.963361 24597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:25.963550 24597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:50:25.972776 24597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:25.974107 24597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:25.975950 24597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:25.988508 24603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:25.988546 24602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:25.989615 24605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:25.990255 24597 server_base.cc:1034] running on GCE node
I20260619 16:50:25.990859 24597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:25.991833 24597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:25.993036 24597 hybrid_clock.cc:648] HybridClock initialized: now 1781887825992971 us; error 60 us; skew 500 ppm
I20260619 16:50:25.993461 24597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:50:25.995916 24597 webserver.cc:460] Webserver started at http://127.14.153.3:42051/ using document root <none> and password file <none>
I20260619 16:50:25.996707 24597 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:25.996825 24597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:25.997190 24597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:26.000131 24597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "a3ef61b10d6e44828861d3073e66a1a8"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-qb36"
server_key: "cfc7bee0d33b7292b156c956d6b2b697"
server_key_iv: "b64f4fe270d404c4c9f52200a46e2b9b"
server_key_version: "encryptionkey@0"
I20260619 16:50:26.000965 24597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "a3ef61b10d6e44828861d3073e66a1a8"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-qb36"
server_key: "cfc7bee0d33b7292b156c956d6b2b697"
server_key_iv: "b64f4fe270d404c4c9f52200a46e2b9b"
server_key_version: "encryptionkey@0"
I20260619 16:50:26.006309 24597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 16:50:26.009886 24611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:26.011260 24597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:50:26.011441 24597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "a3ef61b10d6e44828861d3073e66a1a8"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-qb36"
server_key: "cfc7bee0d33b7292b156c956d6b2b697"
server_key_iv: "b64f4fe270d404c4c9f52200a46e2b9b"
server_key_version: "encryptionkey@0"
I20260619 16:50:26.011652 24597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:26.033910 24597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:26.034801 24597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:26.035138 24597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:26.036103 24597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:26.037598 24597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:26.037711 24597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:26.037842 24597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:26.037932 24597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:26.071774 24597 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:46393
I20260619 16:50:26.071830 24723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:46393 every 8 connection(s)
I20260619 16:50:26.073156 24597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:26.078306 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 24597
I20260619 16:50:26.078536 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:26.079123 14948 external_mini_cluster.cc:1428] Setting key e5ed94caf91158b89b7ce37cfc989cbd
I20260619 16:50:26.086761 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:26.075727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:42631 (local address 127.14.153.62:41447)
0619 16:50:26.075947 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:26.075954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:26.077561 (+ 1607us) server_negotiation.cc:407] Connection header received
0619 16:50:26.080094 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:26.080099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:26.080203 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:26.080311 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:26.081702 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.082577 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.084185 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.084553 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.084644 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:26.086075 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:26.086080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:26.086083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:26.086101 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:26.086130 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:26.086141 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:26.086209 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:26.086445 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 16:50:26.086517 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:50:26.087934 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:26.076364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41447 (local address 127.14.153.3:42631)
0619 16:50:26.077387 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:26.077436 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:26.079744 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:26.080623 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:26.080635 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:26.080765 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:26.081542 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:26.081560 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.082706 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.082710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:26.084065 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:26.084077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.084795 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.084800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:26.085007 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:26.085766 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:26.085769 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:26.085808 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:26.085900 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:26.086224 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:26.086231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:26.086237 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:50:26.086423 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 16:50:26.086733 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":301,"threads_started":1}
I20260619 16:50:26.089272 24724 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41447
I20260619 16:50:26.089596 24724 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:26.090314 24724 heartbeater.cc:511] Master 127.14.153.62:41447 requested a full tablet report, sending...
I20260619 16:50:26.091732 24277 ts_manager.cc:194] Registered new tserver with Master: a3ef61b10d6e44828861d3073e66a1a8 (127.14.153.3:46393)
I20260619 16:50:26.092520 24277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.3:42631
I20260619 16:50:26.094640 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:50:26.100703 24465 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:26.099108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.14.153.62:41447)
0619 16:50:26.099256 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:26.099261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:26.099725 (+ 464us) server_negotiation.cc:407] Connection header received
0619 16:50:26.099890 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:26.099894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:26.099960 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:26.100042 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:26.100457 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55754: BlockingRecv error: recv got EOF from 127.0.0.1:55754 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:50:26.101240 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 24336
I20260619 16:50:26.109385 24733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:26.102119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.14.153.1:44453)
0619 16:50:26.102661 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:26.102670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:26.102752 (+ 82us) server_negotiation.cc:407] Connection header received
0619 16:50:26.102842 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:26.102849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:26.103160 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:26.103341 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:26.103634 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.104639 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.106045 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.106685 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.106786 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:26.107894 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:26.107911 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:26.107924 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:26.108513 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:26.108564 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:26.108584 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:26.108709 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:26.108898 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 16:50:26.109059 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":329,"threads_started":1}
I20260619 16:50:26.110620 24458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:26.189817 24463 heartbeater.cc:503] Master 127.14.153.62:41447 was elected leader, sending a full tablet report...
I20260619 16:50:26.191066 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 24336
W20260619 16:50:26.204296 24263 connection.cc:535] server connection from 127.14.153.1:39165 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:50:26.204609 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 24467
I20260619 16:50:26.212524 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:26.205490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.14.153.2:39081)
0619 16:50:26.206024 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:26.206032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:26.206058 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:50:26.206125 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:26.206132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:26.206348 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:26.206483 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:26.206836 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.208029 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.209413 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.210005 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.210114 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:26.211147 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:26.211157 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:26.211172 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:26.211806 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:26.211856 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:26.211882 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:26.211977 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:26.212112 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 16:50:26.212237 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":286,"threads_started":1}
I20260619 16:50:26.213757 24589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:26.294781 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 24467
I20260619 16:50:26.312315 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 24597
I20260619 16:50:26.320494 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:26.313269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.14.153.3:46393)
0619 16:50:26.313819 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:26.313827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:26.313857 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:50:26.313920 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:26.313926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:26.314156 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:26.314341 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:26.314791 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.315938 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.317186 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:26.317766 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:26.317905 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:26.318986 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:26.318994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:26.319014 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:26.319659 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:26.319709 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:26.319729 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:26.319857 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:26.320023 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 16:50:26.320170 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":274,"threads_started":1}
I20260619 16:50:26.321712 24719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:26.404286 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 24597
I20260619 16:50:26.420303 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 24246
I20260619 16:50:26.421572 24307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:26.521641 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 24246
2026-06-19T16:50:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2453 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T16:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:26Z Disabled control of system clock
I20260619 16:50:26.592105 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:41987
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:41987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:50:26.918402 24743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:26.918705 24743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:26.918790 24743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:26.926147 24743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:26.926260 24743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:26.926298 24743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:26.926362 24743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:26.926398 24743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:26.936344 24743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:41987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:41987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:26.937651 24743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:26.939405 24743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:26.951242 24751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:26.951541 24748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:26.951278 24749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:26.952528 24743 server_base.cc:1034] running on GCE node
I20260619 16:50:26.953418 24743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:26.954941 24743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:26.956223 24743 hybrid_clock.cc:648] HybridClock initialized: now 1781887826956159 us; error 58 us; skew 500 ppm
I20260619 16:50:26.956629 24743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:50:26.959106 24743 webserver.cc:460] Webserver started at http://127.14.153.62:34223/ using document root <none> and password file <none>
I20260619 16:50:26.959842 24743 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:26.959978 24743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:26.960307 24743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:26.963572 24743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "f70f6ecaa13a4705b5d5452606e6ea40"
format_stamp: "Formatted at 2026-06-19 16:50:26 on dist-test-slave-qb36"
server_key: "23140339307306fe765cea456b4944c6"
server_key_iv: "74aa3225c50c69e53e3717f0b967c866"
server_key_version: "encryptionkey@0"
I20260619 16:50:26.964360 24743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "f70f6ecaa13a4705b5d5452606e6ea40"
format_stamp: "Formatted at 2026-06-19 16:50:26 on dist-test-slave-qb36"
server_key: "23140339307306fe765cea456b4944c6"
server_key_iv: "74aa3225c50c69e53e3717f0b967c866"
server_key_version: "encryptionkey@0"
I20260619 16:50:26.969388 24743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:50:26.972820 24757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:26.974150 24743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:50:26.974334 24743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "f70f6ecaa13a4705b5d5452606e6ea40"
format_stamp: "Formatted at 2026-06-19 16:50:26 on dist-test-slave-qb36"
server_key: "23140339307306fe765cea456b4944c6"
server_key_iv: "74aa3225c50c69e53e3717f0b967c866"
server_key_version: "encryptionkey@0"
I20260619 16:50:26.974566 24743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:26.980324 24743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:26.981113 24743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:26.981411 24743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:26.999660 24743 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:41987
I20260619 16:50:26.999677 24808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:41987 every 8 connection(s)
I20260619 16:50:27.001073 24743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:27.003273 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 24743
I20260619 16:50:27.003535 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:50:27.004350 14948 external_mini_cluster.cc:1428] Setting key 093e29131a592cd45c76c06f41636eec
I20260619 16:50:27.005950 24809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:27.012885 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:27.006552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.14.153.62:41987)
0619 16:50:27.007592 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:27.007603 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:27.007643 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:50:27.009179 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:27.009202 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:27.009638 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:27.010076 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:27.011205 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:27.011211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:27.011227 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:27.011245 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:27.011316 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:27.011335 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:27.011573 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:27.011710 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 16:50:27.011971 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":381,"threads_started":1}
I20260619 16:50:27.018268 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40: Bootstrap starting.
I20260619 16:50:27.021714 24809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:27.022723 24809 log.cc:824] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:27.025478 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40: No bootstrap required, opened a new log
I20260619 16:50:27.033563 24809 raft_consensus.cc:348] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70f6ecaa13a4705b5d5452606e6ea40" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41987 } }
I20260619 16:50:27.034049 24809 raft_consensus.cc:374] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:27.034163 24809 raft_consensus.cc:729] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f70f6ecaa13a4705b5d5452606e6ea40, State: Initialized, Role: FOLLOWER
I20260619 16:50:27.034809 24809 consensus_queue.cc:260] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70f6ecaa13a4705b5d5452606e6ea40" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41987 } }
I20260619 16:50:27.035012 24809 raft_consensus.cc:388] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:27.035185 24809 raft_consensus.cc:482] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:27.035306 24809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:27.037474 24809 raft_consensus.cc:504] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70f6ecaa13a4705b5d5452606e6ea40" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41987 } }
I20260619 16:50:27.037914 24809 leader_election.cc:302] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f70f6ecaa13a4705b5d5452606e6ea40; no voters:
I20260619 16:50:27.038416 24809 leader_election.cc:288] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:27.038573 24814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:27.039288 24814 raft_consensus.cc:686] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [term 1 LEADER]: Becoming Leader. State: Replica: f70f6ecaa13a4705b5d5452606e6ea40, State: Running, Role: LEADER
I20260619 16:50:27.039870 24814 consensus_queue.cc:237] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70f6ecaa13a4705b5d5452606e6ea40" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41987 } }
I20260619 16:50:27.040537 24809 sys_catalog.cc:564] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:27.042740 24816 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f70f6ecaa13a4705b5d5452606e6ea40. Latest consensus state: current_term: 1 leader_uuid: "f70f6ecaa13a4705b5d5452606e6ea40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70f6ecaa13a4705b5d5452606e6ea40" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41987 } } }
I20260619 16:50:27.043226 24816 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:27.044485 24815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f70f6ecaa13a4705b5d5452606e6ea40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70f6ecaa13a4705b5d5452606e6ea40" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41987 } } }
I20260619 16:50:27.044997 24815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:27.045867 24822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:27.050900 24822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:27.059275 24822 catalog_manager.cc:1347] Generated new cluster ID: 8ede29915ca5404a9d2822c1dfb6e3a8
I20260619 16:50:27.059419 24822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:27.087157 24822 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:27.087652 24822 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:27.094441 24822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f70f6ecaa13a4705b5d5452606e6ea40: Generated new TSK 0
I20260619 16:50:27.095484 24822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:27.111375 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41987
--builtin_ntp_servers=127.14.153.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:50:27.445681 24833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:27.446024 24833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:27.446208 24833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:27.453594 24833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:27.453738 24833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:27.453929 24833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:50:27.463294 24833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:27.464645 24833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:27.466368 24833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:27.479696 24839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:27.479780 24841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:27.479820 24838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:27.484526 24833 server_base.cc:1034] running on GCE node
I20260619 16:50:27.485196 24833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:27.486105 24833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:27.487301 24833 hybrid_clock.cc:648] HybridClock initialized: now 1781887827487248 us; error 44 us; skew 500 ppm
I20260619 16:50:27.487598 24833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:50:27.489843 24833 webserver.cc:460] Webserver started at http://127.14.153.1:41855/ using document root <none> and password file <none>
I20260619 16:50:27.490496 24833 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:27.490579 24833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:27.490851 24833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:27.493844 24833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "dfa04a0c56d3473dbf64d30b43b3ea64"
format_stamp: "Formatted at 2026-06-19 16:50:27 on dist-test-slave-qb36"
server_key: "a5143ce8ce781ed5b4a68e0c685c079a"
server_key_iv: "ddce6eaaf2596a89385c027b40add900"
server_key_version: "encryptionkey@0"
I20260619 16:50:27.494599 24833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "dfa04a0c56d3473dbf64d30b43b3ea64"
format_stamp: "Formatted at 2026-06-19 16:50:27 on dist-test-slave-qb36"
server_key: "a5143ce8ce781ed5b4a68e0c685c079a"
server_key_iv: "ddce6eaaf2596a89385c027b40add900"
server_key_version: "encryptionkey@0"
I20260619 16:50:27.499725 24833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:50:27.502995 24847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:27.504496 24833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:50:27.504632 24833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "dfa04a0c56d3473dbf64d30b43b3ea64"
format_stamp: "Formatted at 2026-06-19 16:50:27 on dist-test-slave-qb36"
server_key: "a5143ce8ce781ed5b4a68e0c685c079a"
server_key_iv: "ddce6eaaf2596a89385c027b40add900"
server_key_version: "encryptionkey@0"
I20260619 16:50:27.504819 24833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:27.510409 24833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:27.511242 24833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:27.511468 24833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:27.512430 24833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:27.513756 24833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:27.513840 24833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:27.513923 24833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:27.513978 24833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:27.547580 24833 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:41155
I20260619 16:50:27.547628 24959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:41155 every 8 connection(s)
I20260619 16:50:27.548897 24833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:27.554154 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 24833
I20260619 16:50:27.554356 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:27.555112 14948 external_mini_cluster.cc:1428] Setting key 8f3e16c2e45234ff9e8ca42642762db0
I20260619 16:50:27.558159 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:27.551394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:54537 (local address 127.14.153.62:41987)
0619 16:50:27.551958 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:27.551963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:27.553049 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 16:50:27.555370 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:27.555375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:27.555426 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:27.555527 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:27.557167 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:27.557173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:27.557179 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:27.557200 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:27.557249 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:27.557267 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:27.557409 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:27.557738 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 16:50:27.557806 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":257,"threads_started":1}
I20260619 16:50:27.558741 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41987
--builtin_ntp_servers=127.14.153.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:50:27.559319 24963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:27.552052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41987 (local address 127.14.153.1:54537)
0619 16:50:27.552898 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:27.552935 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:27.555067 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:27.555792 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:27.555800 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:27.555922 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:27.556791 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:27.556794 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:27.556851 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:27.556963 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:27.557475 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:27.557489 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:27.557498 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:50:27.557783 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 16:50:27.558050 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":241,"threads_started":1}
I20260619 16:50:27.561139 24960 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41987
I20260619 16:50:27.561510 24960 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:27.562229 24960 heartbeater.cc:511] Master 127.14.153.62:41987 requested a full tablet report, sending...
I20260619 16:50:27.564821 24774 ts_manager.cc:194] Registered new tserver with Master: dfa04a0c56d3473dbf64d30b43b3ea64 (127.14.153.1:41155)
W20260619 16:50:27.893500 24964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:27.893803 24964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:27.893893 24964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:27.901772 24964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:27.901878 24964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:27.901988 24964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:50:27.911386 24964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:27.912649 24964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:27.914359 24964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:27.927623 24969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:27.927585 24972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:27.927542 24970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:27.928553 24964 server_base.cc:1034] running on GCE node
I20260619 16:50:27.929157 24964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:27.930145 24964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:27.931363 24964 hybrid_clock.cc:648] HybridClock initialized: now 1781887827931313 us; error 66 us; skew 500 ppm
I20260619 16:50:27.931694 24964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:50:27.933884 24964 webserver.cc:460] Webserver started at http://127.14.153.2:39953/ using document root <none> and password file <none>
I20260619 16:50:27.934605 24964 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:27.934738 24964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:27.935124 24964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:27.938010 24964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "4126d447f34649e8b548037c4c19cd06"
format_stamp: "Formatted at 2026-06-19 16:50:27 on dist-test-slave-qb36"
server_key: "f02693397d7ca468cdb96ed1a55ad426"
server_key_iv: "e856104b00c56c049a371520341ca446"
server_key_version: "encryptionkey@0"
I20260619 16:50:27.938848 24964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "4126d447f34649e8b548037c4c19cd06"
format_stamp: "Formatted at 2026-06-19 16:50:27 on dist-test-slave-qb36"
server_key: "f02693397d7ca468cdb96ed1a55ad426"
server_key_iv: "e856104b00c56c049a371520341ca446"
server_key_version: "encryptionkey@0"
I20260619 16:50:27.944211 24964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:50:27.947992 24978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:27.949352 24964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:50:27.949532 24964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "4126d447f34649e8b548037c4c19cd06"
format_stamp: "Formatted at 2026-06-19 16:50:27 on dist-test-slave-qb36"
server_key: "f02693397d7ca468cdb96ed1a55ad426"
server_key_iv: "e856104b00c56c049a371520341ca446"
server_key_version: "encryptionkey@0"
I20260619 16:50:27.949764 24964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:27.955411 24964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:27.956219 24964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:27.956494 24964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:27.957409 24964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:27.958786 24964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:27.958912 24964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:27.959079 24964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:27.959177 24964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:50:27.992265 24964 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:34433
I20260619 16:50:27.992375 25090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:34433 every 8 connection(s)
I20260619 16:50:27.993654 24964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:28.002013 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:27.996247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:50129 (local address 127.14.153.62:41987)
0619 16:50:27.996419 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:27.996424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:27.997873 (+ 1449us) server_negotiation.cc:407] Connection header received
0619 16:50:27.999684 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:27.999690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:27.999767 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:27.999877 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:28.001255 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:28.001260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:28.001263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:28.001278 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:28.001310 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:28.001321 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:28.001453 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:28.001652 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 16:50:28.001726 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:50:28.002779 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 24964
I20260619 16:50:28.002970 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:28.003099 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:27.996795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41987 (local address 127.14.153.2:50129)
0619 16:50:27.997700 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:27.997742 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:27.999394 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:28.000112 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:28.000121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:28.000221 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:28.000938 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:28.000941 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:28.000981 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:28.001065 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:28.001415 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:28.001426 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:28.001434 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:50:28.001739 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 16:50:28.002005 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":261,"threads_started":1}
I20260619 16:50:28.003494 14948 external_mini_cluster.cc:1428] Setting key da0cb91357568e42e79344fb8f70fe0c
I20260619 16:50:28.005636 25091 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41987
I20260619 16:50:28.005970 25091 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:28.006620 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41987
--builtin_ntp_servers=127.14.153.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:50:28.006631 25091 heartbeater.cc:511] Master 127.14.153.62:41987 requested a full tablet report, sending...
I20260619 16:50:28.008364 24774 ts_manager.cc:194] Registered new tserver with Master: 4126d447f34649e8b548037c4c19cd06 (127.14.153.2:34433)
W20260619 16:50:28.335734 25094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:28.336011 25094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:28.336179 25094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:28.343727 25094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:28.343856 25094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:28.344040 25094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:50:28.353435 25094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:28.354800 25094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:28.356604 25094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:28.370020 25102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:28.370043 25100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:28.370507 25099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:28.370851 25094 server_base.cc:1034] running on GCE node
I20260619 16:50:28.371549 25094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:28.372501 25094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:28.373739 25094 hybrid_clock.cc:648] HybridClock initialized: now 1781887828373692 us; error 69 us; skew 500 ppm
I20260619 16:50:28.374063 25094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:50:28.376446 25094 webserver.cc:460] Webserver started at http://127.14.153.3:44089/ using document root <none> and password file <none>
I20260619 16:50:28.377157 25094 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:28.377295 25094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:28.377624 25094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:28.380617 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "ff297841d2594e8c9086cfd928cd8724"
format_stamp: "Formatted at 2026-06-19 16:50:28 on dist-test-slave-qb36"
server_key: "2a2bb955db400027feb7ce99537cce5b"
server_key_iv: "f88e9d572135c7047f9b86fdebe9e0a1"
server_key_version: "encryptionkey@0"
I20260619 16:50:28.381402 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "ff297841d2594e8c9086cfd928cd8724"
format_stamp: "Formatted at 2026-06-19 16:50:28 on dist-test-slave-qb36"
server_key: "2a2bb955db400027feb7ce99537cce5b"
server_key_iv: "f88e9d572135c7047f9b86fdebe9e0a1"
server_key_version: "encryptionkey@0"
I20260619 16:50:28.386592 25094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 16:50:28.390021 25108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:28.391407 25094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:28.391599 25094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "ff297841d2594e8c9086cfd928cd8724"
format_stamp: "Formatted at 2026-06-19 16:50:28 on dist-test-slave-qb36"
server_key: "2a2bb955db400027feb7ce99537cce5b"
server_key_iv: "f88e9d572135c7047f9b86fdebe9e0a1"
server_key_version: "encryptionkey@0"
I20260619 16:50:28.391813 25094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:28.397375 25094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:28.398185 25094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:28.398468 25094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:28.399444 25094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:28.400876 25094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:28.400992 25094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:28.401106 25094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:28.401194 25094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:28.433949 25094 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:45425
I20260619 16:50:28.434005 25220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:45425 every 8 connection(s)
I20260619 16:50:28.435312 25094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:28.442056 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 25094
I20260619 16:50:28.442265 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:28.442821 14948 external_mini_cluster.cc:1428] Setting key 0001937ff16a2a0dd49de4b37956e471
I20260619 16:50:28.443944 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:28.437978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:53541 (local address 127.14.153.62:41987)
0619 16:50:28.438162 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:28.438166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:28.439665 (+ 1499us) server_negotiation.cc:407] Connection header received
0619 16:50:28.441390 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:28.441395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:28.441466 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:28.441565 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:28.443215 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:28.443220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:28.443223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:28.443238 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:28.443277 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:28.443288 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:28.443414 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:28.443610 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 16:50:28.443678 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:50:28.445264 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:28.438521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41987 (local address 127.14.153.3:53541)
0619 16:50:28.439497 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:28.439547 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:28.441106 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:28.441898 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:28.441934 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:28.442078 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:28.442823 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:28.442826 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:28.442883 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:28.443019 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:28.443395 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:28.443415 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:28.443422 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:50:28.443682 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 16:50:28.443977 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":260,"threads_started":1}
I20260619 16:50:28.446748 25221 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41987
I20260619 16:50:28.447003 25221 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:28.447642 25221 heartbeater.cc:511] Master 127.14.153.62:41987 requested a full tablet report, sending...
I20260619 16:50:28.448983 24774 ts_manager.cc:194] Registered new tserver with Master: ff297841d2594e8c9086cfd928cd8724 (127.14.153.3:45425)
I20260619 16:50:28.458681 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:50:28.465054 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:28.463390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.14.153.62:41987)
0619 16:50:28.463662 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:28.463670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:28.464065 (+ 395us) server_negotiation.cc:407] Connection header received
0619 16:50:28.464300 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:28.464304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:28.464355 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:28.464463 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:28.464796 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38256: BlockingRecv error: recv got EOF from 127.0.0.1:38256 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260619 16:50:28.465934 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 24833
I20260619 16:50:28.469306 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:28.466829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.14.153.1:41155)
0619 16:50:28.467362 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:28.467373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:28.467405 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:50:28.467462 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:28.467469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:28.467697 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:28.467833 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:28.468693 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:28.468699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:28.468710 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:28.468726 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:28.468773 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:28.468787 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:28.468890 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:28.469007 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 16:50:28.469081 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
I20260619 16:50:28.470543 24955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:28.552867 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 24833
I20260619 16:50:28.569154 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 24964
I20260619 16:50:28.573009 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:28.570064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.14.153.2:34433)
0619 16:50:28.570625 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:28.570632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:28.570658 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:50:28.570719 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:28.570726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:28.570951 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:28.571143 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:28.572246 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:28.572253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:28.572265 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:28.572327 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:28.572373 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:28.572388 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:28.572552 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:28.572623 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 16:50:28.572734 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":257,"threads_started":1}
I20260619 16:50:28.574353 25086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:28.655335 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 24964
I20260619 16:50:28.671713 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 25094
I20260619 16:50:28.675849 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:28.672832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.14.153.3:45425)
0619 16:50:28.673489 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:28.673497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:28.673526 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:50:28.673588 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:28.673594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:28.673809 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:28.673956 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:28.674999 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:28.675005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:28.675019 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:28.675089 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:28.675149 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:28.675168 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:28.675302 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:28.675518 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 16:50:28.675603 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":308,"threads_started":1}
I20260619 16:50:28.677145 25216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:28.759200 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 25094
I20260619 16:50:28.775523 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 24743
I20260619 16:50:28.776737 24804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:28.879796 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 24743
2026-06-19T16:50:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2357 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:28 dist-test-slave-qb36 krb5kdc[25239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:28 dist-test-slave-qb36 krb5kdc[25239](info): set up 2 sockets
Jun 19 16:50:28 dist-test-slave-qb36 krb5kdc[25239](info): commencing operation
krb5kdc: starting...
W20260619 16:50:29.996071 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:50:30 dist-test-slave-qb36 krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:50:30.251682 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35331
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:30.578847 25252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:30.579299 25252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:30.579466 25252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:30.586869 25252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:30.587040 25252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:30.587175 25252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:30.587284 25252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:30.587360 25252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:30.597548 25252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:30.598976 25252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:30.600404 25252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:30.612504 25258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:30.612525 25257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:30.613245 25252 server_base.cc:1034] running on GCE node
W20260619 16:50:30.613360 25260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:30.614205 25252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:30.615808 25252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:30.617048 25252 hybrid_clock.cc:648] HybridClock initialized: now 1781887830617006 us; error 40 us; skew 500 ppm
I20260619 16:50:30.617440 25252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:30 dist-test-slave-qb36 krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887830, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:30.620620 25252 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:50:30.622128 25252 webserver.cc:460] Webserver started at http://127.14.153.62:33525/ using document root <none> and password file <none>
I20260619 16:50:30.622836 25252 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:30.622982 25252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:30.623386 25252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:30.626324 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "d42d52e5280940459859d839ca6fe688"
format_stamp: "Formatted at 2026-06-19 16:50:30 on dist-test-slave-qb36"
server_key: "ccbc176d315c5e7fea490e937f0605c5"
server_key_iv: "f51cc9064b22fb538f96c8f9a671f1ad"
server_key_version: "encryptionkey@0"
I20260619 16:50:30.627177 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "d42d52e5280940459859d839ca6fe688"
format_stamp: "Formatted at 2026-06-19 16:50:30 on dist-test-slave-qb36"
server_key: "ccbc176d315c5e7fea490e937f0605c5"
server_key_iv: "f51cc9064b22fb538f96c8f9a671f1ad"
server_key_version: "encryptionkey@0"
I20260619 16:50:30.632261 25252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:50:30.635710 25267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:30.637033 25252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:50:30.637218 25252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "d42d52e5280940459859d839ca6fe688"
format_stamp: "Formatted at 2026-06-19 16:50:30 on dist-test-slave-qb36"
server_key: "ccbc176d315c5e7fea490e937f0605c5"
server_key_iv: "f51cc9064b22fb538f96c8f9a671f1ad"
server_key_version: "encryptionkey@0"
I20260619 16:50:30.637434 25252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:30.650089 25252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:30.654497 25252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:30.654783 25252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:30.673187 25252 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35331
I20260619 16:50:30.673209 25318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35331 every 8 connection(s)
I20260619 16:50:30.674638 25252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:30.678436 25319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:30.684665 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 25252
I20260619 16:50:30.684909 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:50:30.685684 14948 external_mini_cluster.cc:1428] Setting key e6963d471b767455c06324b9552c2fef
I20260619 16:50:30.690212 25319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688: Bootstrap starting.
I20260619 16:50:30.694027 25319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:30.695267 25319 log.cc:824] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:30.698624 25319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688: No bootstrap required, opened a new log
Jun 19 16:50:30 dist-test-slave-qb36 krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:50:30.706671 25319 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42d52e5280940459859d839ca6fe688" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35331 } }
I20260619 16:50:30.707077 25319 raft_consensus.cc:374] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:30.707224 25319 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42d52e5280940459859d839ca6fe688, State: Initialized, Role: FOLLOWER
I20260619 16:50:30.708006 25319 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42d52e5280940459859d839ca6fe688" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35331 } }
I20260619 16:50:30.708268 25319 raft_consensus.cc:388] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:30.708451 25319 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:30.708639 25319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:30.709263 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:30.687914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.14.153.62:35331)
0619 16:50:30.688744 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:30.688757 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:30.688800 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:50:30.690514 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:30.690542 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:30.690941 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:30.691334 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:30.693223 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:30.694568 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:30.696347 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:30.697134 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:30.697279 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:30.701813 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:30.701821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:30.701840 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:30.701878 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:30.705044 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:30.705745 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:30.705752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:30.705759 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:30.705927 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:30.706312 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:30.706321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:30.706323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:30.706843 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:30.707184 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:30.707604 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 16:50:30.708019 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":400,"threads_started":1}
I20260619 16:50:30.711138 25319 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42d52e5280940459859d839ca6fe688" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35331 } }
I20260619 16:50:30.711663 25319 leader_election.cc:302] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42d52e5280940459859d839ca6fe688; no voters:
I20260619 16:50:30.712383 25319 leader_election.cc:288] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:30.712448 25324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:30.713194 25324 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [term 1 LEADER]: Becoming Leader. State: Replica: d42d52e5280940459859d839ca6fe688, State: Running, Role: LEADER
I20260619 16:50:30.713804 25324 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42d52e5280940459859d839ca6fe688" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35331 } }
I20260619 16:50:30.714480 25319 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:30.716284 25325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d42d52e5280940459859d839ca6fe688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42d52e5280940459859d839ca6fe688" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35331 } } }
I20260619 16:50:30.716639 25325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:30.717344 25326 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42d52e5280940459859d839ca6fe688. Latest consensus state: current_term: 1 leader_uuid: "d42d52e5280940459859d839ca6fe688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42d52e5280940459859d839ca6fe688" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35331 } } }
I20260619 16:50:30.717715 25326 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:30.717993 25330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:30.723143 25330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:30.736502 25330 catalog_manager.cc:1347] Generated new cluster ID: 8583cce3daa042b4ab97ddeca2f0d5db
I20260619 16:50:30.736657 25330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:30.751430 25330 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:30.753259 25330 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:30.772207 25330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688: Generated new TSK 0
I20260619 16:50:30.773294 25330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:50:30.834231 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35331
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:31.166705 25345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:31.166975 25345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:31.167141 25345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:31.174234 25345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:31.174388 25345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:31.174523 25345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:50:31.184657 25345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:31.186008 25345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:31.187276 25345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:31.200358 25351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:31.200274 25353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:31.200310 25350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:31.204833 25345 server_base.cc:1034] running on GCE node
I20260619 16:50:31.205611 25345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:31.206597 25345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:31.207837 25345 hybrid_clock.cc:648] HybridClock initialized: now 1781887831207781 us; error 56 us; skew 500 ppm
I20260619 16:50:31.208177 25345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:31 dist-test-slave-qb36 krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887831, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:31.211634 25345 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:50:31.213156 25345 webserver.cc:460] Webserver started at http://127.14.153.1:45493/ using document root <none> and password file <none>
I20260619 16:50:31.213867 25345 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:31.213966 25345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:31.214205 25345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:31.217374 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "a257fa8390124bb683d225a3f61b1e3e"
format_stamp: "Formatted at 2026-06-19 16:50:31 on dist-test-slave-qb36"
server_key: "f4fb4ca563b2a24d4981648fe054b7a6"
server_key_iv: "5670e7db4d5378557ed0cca115a57b87"
server_key_version: "encryptionkey@0"
I20260619 16:50:31.218101 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "a257fa8390124bb683d225a3f61b1e3e"
format_stamp: "Formatted at 2026-06-19 16:50:31 on dist-test-slave-qb36"
server_key: "f4fb4ca563b2a24d4981648fe054b7a6"
server_key_iv: "5670e7db4d5378557ed0cca115a57b87"
server_key_version: "encryptionkey@0"
I20260619 16:50:31.223306 25345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 16:50:31.226725 25360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:31.228183 25345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:31.228319 25345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "a257fa8390124bb683d225a3f61b1e3e"
format_stamp: "Formatted at 2026-06-19 16:50:31 on dist-test-slave-qb36"
server_key: "f4fb4ca563b2a24d4981648fe054b7a6"
server_key_iv: "5670e7db4d5378557ed0cca115a57b87"
server_key_version: "encryptionkey@0"
I20260619 16:50:31.228495 25345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:31.249712 25345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:31.254144 25345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:31.254419 25345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:31.255396 25345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:31.256738 25345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:31.256814 25345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:31.256896 25345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:31.256942 25345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:31.290673 25345 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:36541
I20260619 16:50:31.290781 25472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:36541 every 8 connection(s)
I20260619 16:50:31.292009 25345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:31.297276 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 25345
I20260619 16:50:31.297506 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:31.298116 14948 external_mini_cluster.cc:1428] Setting key ded1668f4998886763ab4ea5ca7e9d8c
Jun 19 16:50:31 dist-test-slave-qb36 krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887831, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:50:31.313068 25475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:31.294572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:41663 (local address 127.14.153.62:35331)
0619 16:50:31.294994 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:31.294998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:31.295946 (+ 948us) server_negotiation.cc:407] Connection header received
0619 16:50:31.297830 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:31.297836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:31.297928 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:31.298045 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:31.299971 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.300985 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.302601 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.303309 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.303458 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:31.307012 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:31.307021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:31.307057 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:31.307104 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:31.309687 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:31.310699 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:31.310703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:31.310705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:31.310816 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:31.311330 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:31.311333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:31.311335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:31.311600 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:31.311716 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:31.312649 (+ 933us) server_negotiation.cc:299] Negotiation successful
0619 16:50:31.312817 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":233,"threads_started":1}
I20260619 16:50:31.314746 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:31.295090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35331 (local address 127.14.153.1:41663)
0619 16:50:31.295778 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:31.295825 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:31.297535 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:31.298321 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:31.298330 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:31.298982 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:31.299767 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:31.299794 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.301159 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.301163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:31.302444 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:31.302456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.303515 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.303520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:31.303712 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:31.304403 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:31.304426 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:31.306768 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:31.309853 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:31.309866 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:31.309890 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:31.310569 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:31.310925 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:31.310932 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:31.310939 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:31.311213 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:31.311733 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:31.311744 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:31.312225 (+ 481us) client_negotiation.cc:769] Sending connection context
0619 16:50:31.312544 (+ 319us) client_negotiation.cc:240] Negotiation successful
0619 16:50:31.313175 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":242,"threads_started":1}
I20260619 16:50:31.316350 25473 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35331
I20260619 16:50:31.316710 25473 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:31.317668 25473 heartbeater.cc:511] Master 127.14.153.62:35331 requested a full tablet report, sending...
I20260619 16:50:31.320341 25284 ts_manager.cc:194] Registered new tserver with Master: a257fa8390124bb683d225a3f61b1e3e (127.14.153.1:36541)
I20260619 16:50:31.322593 25284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:41663
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:50:31.354846 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35331
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:31.679397 25479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:31.679739 25479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:31.679915 25479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:31.687804 25479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:31.687968 25479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:31.688138 25479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:50:31.698015 25479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:31.699445 25479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:31.700730 25479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:31.714596 25484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:31.714592 25487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:31.714592 25485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:31.715579 25479 server_base.cc:1034] running on GCE node
I20260619 16:50:31.716205 25479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:31.717095 25479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:31.718323 25479 hybrid_clock.cc:648] HybridClock initialized: now 1781887831718270 us; error 59 us; skew 500 ppm
I20260619 16:50:31.718657 25479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:31 dist-test-slave-qb36 krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887831, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:31.721933 25479 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:50:31.723445 25479 webserver.cc:460] Webserver started at http://127.14.153.2:44691/ using document root <none> and password file <none>
I20260619 16:50:31.724138 25479 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:31.724277 25479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:31.724572 25479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:31.727588 25479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "9c225cacbd694563bcacfbeaed3df0c2"
format_stamp: "Formatted at 2026-06-19 16:50:31 on dist-test-slave-qb36"
server_key: "858b040bfa654474b47c525056a52c63"
server_key_iv: "3b8a923cf90cbc7f5116a6cd207093d5"
server_key_version: "encryptionkey@0"
I20260619 16:50:31.728356 25479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "9c225cacbd694563bcacfbeaed3df0c2"
format_stamp: "Formatted at 2026-06-19 16:50:31 on dist-test-slave-qb36"
server_key: "858b040bfa654474b47c525056a52c63"
server_key_iv: "3b8a923cf90cbc7f5116a6cd207093d5"
server_key_version: "encryptionkey@0"
I20260619 16:50:31.733762 25479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:50:31.737517 25494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:31.738955 25479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 16:50:31.739156 25479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "9c225cacbd694563bcacfbeaed3df0c2"
format_stamp: "Formatted at 2026-06-19 16:50:31 on dist-test-slave-qb36"
server_key: "858b040bfa654474b47c525056a52c63"
server_key_iv: "3b8a923cf90cbc7f5116a6cd207093d5"
server_key_version: "encryptionkey@0"
I20260619 16:50:31.739355 25479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:31.767252 25479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:31.771746 25479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:31.772070 25479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:31.773169 25479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:31.774525 25479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:31.774633 25479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:31.774775 25479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:31.774847 25479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:31.811460 25479 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:38373
I20260619 16:50:31.811513 25606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:38373 every 8 connection(s)
I20260619 16:50:31.812754 25479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:31.818429 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 25479
I20260619 16:50:31.818634 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:31.819188 14948 external_mini_cluster.cc:1428] Setting key afa12e21d04f6e5e9e56787a7c8f0649
Jun 19 16:50:31 dist-test-slave-qb36 krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887831, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:50:31.835783 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:31.815490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:41979 (local address 127.14.153.62:35331)
0619 16:50:31.815948 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:31.815954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:31.816992 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 16:50:31.819096 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:31.819101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:31.819166 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:31.819279 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:31.821139 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.822201 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.823900 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.824624 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.824755 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:31.828224 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:31.828234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:31.828240 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:31.828282 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:31.831990 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:31.832894 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:31.832903 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:31.832909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:31.833059 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:31.833608 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:31.833616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:31.833622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:31.833985 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:31.834132 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:31.835007 (+ 875us) server_negotiation.cc:299] Negotiation successful
0619 16:50:31.835277 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
I20260619 16:50:31.836525 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:31.816165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35331 (local address 127.14.153.2:41979)
0619 16:50:31.816821 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:31.816857 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:31.818748 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:31.819547 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:31.819560 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:31.820287 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:31.820940 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:31.820960 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.822358 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.822362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:31.823754 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:31.823766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.824822 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.824827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:31.825015 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:31.825710 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:31.825733 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:31.827977 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:31.832140 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:31.832147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:31.832160 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:31.832756 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:31.833167 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:31.833170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:31.833173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:31.833485 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:31.834141 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:31.834148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:31.834599 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 16:50:31.834895 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 16:50:31.835371 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":250,"threads_started":1}
I20260619 16:50:31.838084 25607 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35331
I20260619 16:50:31.838398 25607 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:31.839187 25607 heartbeater.cc:511] Master 127.14.153.62:35331 requested a full tablet report, sending...
I20260619 16:50:31.840752 25284 ts_manager.cc:194] Registered new tserver with Master: 9c225cacbd694563bcacfbeaed3df0c2 (127.14.153.2:38373)
I20260619 16:50:31.841593 25284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:41979
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:50:31.874173 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35331
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:32.202220 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:32.202539 25613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:32.202638 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:32.210238 25613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:32.210354 25613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:32.210469 25613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:50:32.220706 25613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:50:32.222026 25613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:32.223407 25613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:32.236888 25618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:32.236915 25619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:32.237026 25621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:32.238279 25613 server_base.cc:1034] running on GCE node
I20260619 16:50:32.238903 25613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:32.239918 25613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:32.241195 25613 hybrid_clock.cc:648] HybridClock initialized: now 1781887832241144 us; error 42 us; skew 500 ppm
I20260619 16:50:32.241655 25613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:50:32 dist-test-slave-qb36 krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887832, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:32.245139 25613 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:50:32.246765 25613 webserver.cc:460] Webserver started at http://127.14.153.3:36363/ using document root <none> and password file <none>
I20260619 16:50:32.247565 25613 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:32.247692 25613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:32.248014 25613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:32.251425 25613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "846e453bc1e8464bbf4be4cc3bf3c037"
format_stamp: "Formatted at 2026-06-19 16:50:32 on dist-test-slave-qb36"
server_key: "7b77ba38c43d745da8492d00e3d56f73"
server_key_iv: "7d9e82588772b7d947eebf0f12d0dca0"
server_key_version: "encryptionkey@0"
I20260619 16:50:32.252255 25613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "846e453bc1e8464bbf4be4cc3bf3c037"
format_stamp: "Formatted at 2026-06-19 16:50:32 on dist-test-slave-qb36"
server_key: "7b77ba38c43d745da8492d00e3d56f73"
server_key_iv: "7d9e82588772b7d947eebf0f12d0dca0"
server_key_version: "encryptionkey@0"
I20260619 16:50:32.257521 25613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:50:32.261123 25628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:32.262488 25613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:32.262661 25613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "846e453bc1e8464bbf4be4cc3bf3c037"
format_stamp: "Formatted at 2026-06-19 16:50:32 on dist-test-slave-qb36"
server_key: "7b77ba38c43d745da8492d00e3d56f73"
server_key_iv: "7d9e82588772b7d947eebf0f12d0dca0"
server_key_version: "encryptionkey@0"
I20260619 16:50:32.262856 25613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:32.311205 25613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:32.315724 25613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:32.316027 25613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:32.317059 25613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:32.318475 25613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:32.318578 25613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:32.318727 25613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:32.318796 25613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:32.326162 25473 heartbeater.cc:503] Master 127.14.153.62:35331 was elected leader, sending a full tablet report...
I20260619 16:50:32.353901 25613 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:40983
I20260619 16:50:32.353991 25740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:40983 every 8 connection(s)
I20260619 16:50:32.355271 25613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:32.357710 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 25613
I20260619 16:50:32.357918 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:32.358599 14948 external_mini_cluster.cc:1428] Setting key 515d9012ee175e778263072ac9ff4559
Jun 19 16:50:32 dist-test-slave-qb36 krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887832, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:50:32.375913 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.357838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:41619 (local address 127.14.153.62:35331)
0619 16:50:32.360154 (+ 2316us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.360161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.360182 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:50:32.361845 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.361849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.361981 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.362106 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:32.363620 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.364407 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.365947 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.366322 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.366413 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.370023 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:32.370027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:32.370030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:32.370061 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:32.372713 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.373681 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.373688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.373691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.373835 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.374313 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.374318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.374321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.374632 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:32.374746 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.375486 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.375644 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2199,"thread_start_us":2091,"threads_started":1}
I20260619 16:50:32.376926 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.358412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35331 (local address 127.14.153.3:41619)
0619 16:50:32.359301 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:32.359348 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:32.361544 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:32.362302 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:32.362311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:32.362829 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:32.363468 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.363485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.364536 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.364540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.365776 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.365787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.366496 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.366500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.366698 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.367520 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:32.367543 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:32.369809 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:32.372858 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:32.372867 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:32.372885 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:32.373544 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:32.373950 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:32.373953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:32.373955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:32.374190 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:32.374748 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:32.374754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:32.375150 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 16:50:32.375394 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 16:50:32.375789 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":410,"threads_started":1}
I20260619 16:50:32.378189 25741 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35331
I20260619 16:50:32.378449 25741 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:32.379194 25741 heartbeater.cc:511] Master 127.14.153.62:35331 requested a full tablet report, sending...
I20260619 16:50:32.380474 25284 ts_manager.cc:194] Registered new tserver with Master: 846e453bc1e8464bbf4be4cc3bf3c037 (127.14.153.3:40983)
I20260619 16:50:32.381263 25284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:41619
I20260619 16:50:32.386679 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:50:32.403108 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.391142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.14.153.62:35331)
0619 16:50:32.391349 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.391358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.391711 (+ 353us) server_negotiation.cc:407] Connection header received
0619 16:50:32.391842 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.391845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.391928 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.392007 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:32.393314 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.394023 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.395265 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.395644 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.395733 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.398093 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:32.398097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:32.398101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:32.398130 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:32.400506 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.401130 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.401138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.401146 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.401256 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.401629 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.401635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.401639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.401950 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:32.402205 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.402523 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.402738 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:50:32.407284 25284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:32.414508 25284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:32.451869 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.426854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:38373 (local address 127.0.0.1:49592)
0619 16:50:32.442838 (+ 15984us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:32.442895 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:32.443138 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:32.445425 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:32.445432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:32.445460 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:32.445971 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.445979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.447558 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.447563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.449507 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.449518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.450899 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.450905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.451105 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.451162 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 16:50:32.451261 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:50:32.451452 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15709,"mutex_wait_us":52,"thread_start_us":16094,"threads_started":1}
I20260619 16:50:32.452286 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.426977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.14.153.2:38373)
0619 16:50:32.444874 (+ 17897us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.444881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.444907 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:50:32.444998 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.445006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.445216 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.445419 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:32.446168 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.447411 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.449661 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.450683 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.451558 (+ 875us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.451770 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.451841 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.452030 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17730,"thread_start_us":17793,"threads_started":1}
I20260619 16:50:32.453606 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.430148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.14.153.1:36541)
0619 16:50:32.436632 (+ 6484us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.436646 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.444414 (+ 7768us) server_negotiation.cc:407] Connection header received
0619 16:50:32.444542 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.444549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.444795 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.445002 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:32.445942 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.447551 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.450586 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.452054 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.452195 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.452390 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.453137 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.453293 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6290,"thread_start_us":6129,"threads_started":1}
I20260619 16:50:32.453750 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.426958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:40983 (local address 127.0.0.1:59170)
0619 16:50:32.443752 (+ 16794us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:32.443768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:32.443898 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:32.445020 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:32.445027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:32.445054 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:32.445639 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.445654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.447722 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.447727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.450421 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.450431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.452575 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.452580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.452854 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.452971 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 16:50:32.453059 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:50:32.453380 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16675,"mutex_wait_us":438,"thread_start_us":17043,"threads_started":1}
I20260619 16:50:32.453783 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.427515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.14.153.3:40983)
0619 16:50:32.442745 (+ 15230us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.442754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.444068 (+ 1314us) server_negotiation.cc:407] Connection header received
0619 16:50:32.444189 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.444198 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.444476 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.444682 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:32.445845 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.447551 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.450580 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.452055 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.452177 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.452399 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.453271 (+ 872us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.453438 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15038,"thread_start_us":15235,"threads_started":1}
I20260619 16:50:32.454298 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.429407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:36541 (local address 127.0.0.1:51326)
0619 16:50:32.444282 (+ 14875us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:32.444298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:32.444415 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:32.445060 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:32.445066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:32.445096 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:32.445718 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.445734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.447722 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.447727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.450403 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.450422 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.452560 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.452567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.452853 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.452976 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 16:50:32.453059 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:50:32.453221 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14744,"mutex_wait_us":228,"thread_start_us":14076,"threads_started":1}
I20260619 16:50:32.456377 25542 tablet_service.cc:1467] Processing CreateTablet for tablet dd7e5fa3d3bc4918b1764c053a0d3aa4 (DEFAULT_TABLE table=test-table [id=89a1e49c613a404f855cd11df49b5cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:32.457376 25408 tablet_service.cc:1467] Processing CreateTablet for tablet dd7e5fa3d3bc4918b1764c053a0d3aa4 (DEFAULT_TABLE table=test-table [id=89a1e49c613a404f855cd11df49b5cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:32.457917 25542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7e5fa3d3bc4918b1764c053a0d3aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:32.457406 25676 tablet_service.cc:1467] Processing CreateTablet for tablet dd7e5fa3d3bc4918b1764c053a0d3aa4 (DEFAULT_TABLE table=test-table [id=89a1e49c613a404f855cd11df49b5cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:32.458516 25408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7e5fa3d3bc4918b1764c053a0d3aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:32.458699 25676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7e5fa3d3bc4918b1764c053a0d3aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:32.469383 25760 tablet_bootstrap.cc:492] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: Bootstrap starting.
I20260619 16:50:32.472595 25761 tablet_bootstrap.cc:492] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: Bootstrap starting.
I20260619 16:50:32.474586 25760 tablet_bootstrap.cc:654] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:32.475940 25760 log.cc:824] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:32.476177 25761 tablet_bootstrap.cc:654] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:32.477388 25762 tablet_bootstrap.cc:492] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: Bootstrap starting.
I20260619 16:50:32.477470 25761 log.cc:824] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:32.479285 25760 tablet_bootstrap.cc:492] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: No bootstrap required, opened a new log
I20260619 16:50:32.479552 25760 ts_tablet_manager.cc:1397] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 16:50:32.480741 25761 tablet_bootstrap.cc:492] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: No bootstrap required, opened a new log
I20260619 16:50:32.480810 25762 tablet_bootstrap.cc:654] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:32.480994 25761 ts_tablet_manager.cc:1397] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 16:50:32.482303 25762 log.cc:824] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:32.485292 25762 tablet_bootstrap.cc:492] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: No bootstrap required, opened a new log
I20260619 16:50:32.485546 25762 ts_tablet_manager.cc:1397] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:50:32.487645 25760 raft_consensus.cc:348] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } }
I20260619 16:50:32.488049 25760 raft_consensus.cc:374] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:32.488144 25760 raft_consensus.cc:729] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a257fa8390124bb683d225a3f61b1e3e, State: Initialized, Role: FOLLOWER
I20260619 16:50:32.488874 25760 consensus_queue.cc:260] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } }
I20260619 16:50:32.491761 25761 raft_consensus.cc:348] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } }
I20260619 16:50:32.492261 25761 raft_consensus.cc:374] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:32.492406 25761 raft_consensus.cc:729] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846e453bc1e8464bbf4be4cc3bf3c037, State: Initialized, Role: FOLLOWER
I20260619 16:50:32.492488 25760 ts_tablet_manager.cc:1428] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 16:50:32.493070 25761 consensus_queue.cc:260] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } }
I20260619 16:50:32.494168 25741 heartbeater.cc:503] Master 127.14.153.62:35331 was elected leader, sending a full tablet report...
I20260619 16:50:32.494789 25761 ts_tablet_manager.cc:1428] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260619 16:50:32.497453 25762 raft_consensus.cc:348] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } }
I20260619 16:50:32.497958 25762 raft_consensus.cc:374] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:32.498077 25762 raft_consensus.cc:729] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c225cacbd694563bcacfbeaed3df0c2, State: Initialized, Role: FOLLOWER
I20260619 16:50:32.498762 25762 consensus_queue.cc:260] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } }
I20260619 16:50:32.499754 25607 heartbeater.cc:503] Master 127.14.153.62:35331 was elected leader, sending a full tablet report...
I20260619 16:50:32.500339 25762 ts_tablet_manager.cc:1428] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 16:50:32.502271 25767 raft_consensus.cc:482] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:32.502554 25767 raft_consensus.cc:504] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } }
I20260619 16:50:32.504336 25767 leader_election.cc:288] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c225cacbd694563bcacfbeaed3df0c2 (127.14.153.2:38373), a257fa8390124bb683d225a3f61b1e3e (127.14.153.1:36541)
I20260619 16:50:32.511754 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.505647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:40769 (local address 127.14.153.2:38373)
0619 16:50:32.505879 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.505886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.505904 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:50:32.505966 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.505971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.506044 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.506133 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:32.506599 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.507574 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.509815 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.510860 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.511048 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.511126 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.511371 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.511498 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:50:32.511752 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.504790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:38373 (local address 127.14.153.3:40769)
0619 16:50:32.505211 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:32.505231 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:32.505339 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:32.506183 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:32.506190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:32.506223 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:32.506474 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.506483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.507704 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.507708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.509667 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.509688 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.511022 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.511075 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.511221 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.511245 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:50:32.511321 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:50:32.511442 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"spinlock_wait_cycles":29952}
I20260619 16:50:32.512697 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.504838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:37997 (local address 127.14.153.1:36541)
0619 16:50:32.505045 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.505053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.506281 (+ 1228us) server_negotiation.cc:407] Connection header received
0619 16:50:32.506445 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.506448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.506501 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.506606 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:32.507148 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.508023 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.510228 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.511472 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.511621 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.511686 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.512337 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.512471 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:50:32.512781 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.504813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:36541 (local address 127.14.153.3:37997)
0619 16:50:32.506182 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:32.506198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:32.506326 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:32.506638 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:32.506641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:32.506660 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:32.506973 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.506980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.508168 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.508174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.510091 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:32.510101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.511672 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.511677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:32.512182 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.512207 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:50:32.512298 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:50:32.512418 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"spinlock_wait_cycles":896,"thread_start_us":1040,"threads_started":1}
I20260619 16:50:32.512921 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7e5fa3d3bc4918b1764c053a0d3aa4" candidate_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c225cacbd694563bcacfbeaed3df0c2" is_pre_election: true
I20260619 16:50:32.513479 25562 raft_consensus.cc:2454] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e453bc1e8464bbf4be4cc3bf3c037 in term 0.
I20260619 16:50:32.513707 25428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7e5fa3d3bc4918b1764c053a0d3aa4" candidate_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a257fa8390124bb683d225a3f61b1e3e" is_pre_election: true
I20260619 16:50:32.514231 25632 leader_election.cc:302] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 846e453bc1e8464bbf4be4cc3bf3c037, 9c225cacbd694563bcacfbeaed3df0c2; no voters:
I20260619 16:50:32.514292 25428 raft_consensus.cc:2454] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 846e453bc1e8464bbf4be4cc3bf3c037 in term 0.
I20260619 16:50:32.514678 25767 raft_consensus.cc:2781] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:32.514864 25767 raft_consensus.cc:482] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:32.515007 25767 raft_consensus.cc:3037] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:32.517422 25767 raft_consensus.cc:504] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } }
I20260619 16:50:32.518018 25767 leader_election.cc:288] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [CANDIDATE]: Term 1 election: Requested vote from peers 9c225cacbd694563bcacfbeaed3df0c2 (127.14.153.2:38373), a257fa8390124bb683d225a3f61b1e3e (127.14.153.1:36541)
I20260619 16:50:32.518500 25428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7e5fa3d3bc4918b1764c053a0d3aa4" candidate_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a257fa8390124bb683d225a3f61b1e3e"
I20260619 16:50:32.518499 25562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7e5fa3d3bc4918b1764c053a0d3aa4" candidate_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c225cacbd694563bcacfbeaed3df0c2"
I20260619 16:50:32.518714 25562 raft_consensus.cc:3037] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:32.518714 25428 raft_consensus.cc:3037] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:32.521898 25562 raft_consensus.cc:2454] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846e453bc1e8464bbf4be4cc3bf3c037 in term 1.
I20260619 16:50:32.521896 25428 raft_consensus.cc:2454] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 846e453bc1e8464bbf4be4cc3bf3c037 in term 1.
I20260619 16:50:32.522339 25630 leader_election.cc:302] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 846e453bc1e8464bbf4be4cc3bf3c037, a257fa8390124bb683d225a3f61b1e3e; no voters:
I20260619 16:50:32.522660 25767 raft_consensus.cc:2781] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:32.523113 25767 raft_consensus.cc:686] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 1 LEADER]: Becoming Leader. State: Replica: 846e453bc1e8464bbf4be4cc3bf3c037, State: Running, Role: LEADER
I20260619 16:50:32.523571 25767 consensus_queue.cc:237] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } }
I20260619 16:50:32.528105 25284 catalog_manager.cc:5495] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 reported cstate change: term changed from 0 to 1, leader changed from <none> to 846e453bc1e8464bbf4be4cc3bf3c037 (127.14.153.3). New cstate: current_term: 1 leader_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "846e453bc1e8464bbf4be4cc3bf3c037" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 40983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 } health_report { overall_health: UNKNOWN } } }
W20260619 16:50:32.544274 25474 tablet.cc:2369] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:32.552614 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.547790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.14.153.3:40983)
0619 16:50:32.547995 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.548004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.548110 (+ 106us) server_negotiation.cc:407] Connection header received
0619 16:50:32.548269 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.548273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.548346 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.548489 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:32.548798 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.549508 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.550715 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.551149 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.551239 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.551623 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:32.552104 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:32.552222 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.552262 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.552320 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:50:32.560290 25428 raft_consensus.cc:1261] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [term 1 FOLLOWER]: Refusing update from remote peer 846e453bc1e8464bbf4be4cc3bf3c037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:32.560290 25562 raft_consensus.cc:1261] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [term 1 FOLLOWER]: Refusing update from remote peer 846e453bc1e8464bbf4be4cc3bf3c037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:32.561269 25770 consensus_queue.cc:1035] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a257fa8390124bb683d225a3f61b1e3e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:32.561609 25767 consensus_queue.cc:1035] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c225cacbd694563bcacfbeaed3df0c2" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:50:32.564884 25608 tablet.cc:2369] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:32.592237 25283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:32.592568 25283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:32.595968 25283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d42d52e5280940459859d839ca6fe688: Sending DeleteTablet for 3 replicas of tablet dd7e5fa3d3bc4918b1764c053a0d3aa4
I20260619 16:50:32.597114 25542 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7e5fa3d3bc4918b1764c053a0d3aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:32 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:49592
I20260619 16:50:32.597183 25676 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7e5fa3d3bc4918b1764c053a0d3aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:32 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:59170
I20260619 16:50:32.597786 25408 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7e5fa3d3bc4918b1764c053a0d3aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:32 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:51326
I20260619 16:50:32.600286 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 25345
I20260619 16:50:32.607837 25784 tablet_replica.cc:330] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: stopping tablet replica
I20260619 16:50:32.608300 25784 raft_consensus.cc:2229] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:32.608841 25784 raft_consensus.cc:2258] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:32.612808 25784 ts_tablet_manager.cc:1905] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:32.617369 25782 tablet_replica.cc:330] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: stopping tablet replica
I20260619 16:50:32.617733 25783 tablet_replica.cc:330] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: stopping tablet replica
I20260619 16:50:32.618211 25783 raft_consensus.cc:2229] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:32.618436 25782 raft_consensus.cc:2229] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:32.618757 25783 raft_consensus.cc:2258] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:32.619179 25782 raft_consensus.cc:2258] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:32.621155 25783 ts_tablet_manager.cc:1905] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:32.621204 25782 ts_tablet_manager.cc:1905] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:50:32 dist-test-slave-qb36 krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:50:32.627022 25784 ts_tablet_manager.cc:1918] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:32.628075 25783 ts_tablet_manager.cc:1918] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:32.628285 25783 log.cc:1196] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/dd7e5fa3d3bc4918b1764c053a0d3aa4
I20260619 16:50:32.628860 25783 ts_tablet_manager.cc:1939] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 9c225cacbd694563bcacfbeaed3df0c2: Deleting consensus metadata
I20260619 16:50:32.629089 25782 ts_tablet_manager.cc:1918] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:32.629412 25784 log.cc:1196] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/dd7e5fa3d3bc4918b1764c053a0d3aa4
I20260619 16:50:32.629732 25782 log.cc:1196] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/dd7e5fa3d3bc4918b1764c053a0d3aa4
I20260619 16:50:32.630820 25784 ts_tablet_manager.cc:1939] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P a257fa8390124bb683d225a3f61b1e3e: Deleting consensus metadata
I20260619 16:50:32.630999 25271 catalog_manager.cc:4841] TS 9c225cacbd694563bcacfbeaed3df0c2 (127.14.153.2:38373): tablet dd7e5fa3d3bc4918b1764c053a0d3aa4 (table test-table [id=89a1e49c613a404f855cd11df49b5cc7]) successfully deleted
I20260619 16:50:32.631372 25782 ts_tablet_manager.cc:1939] T dd7e5fa3d3bc4918b1764c053a0d3aa4 P 846e453bc1e8464bbf4be4cc3bf3c037: Deleting consensus metadata
I20260619 16:50:32.632673 25269 catalog_manager.cc:4841] TS a257fa8390124bb683d225a3f61b1e3e (127.14.153.1:36541): tablet dd7e5fa3d3bc4918b1764c053a0d3aa4 (table test-table [id=89a1e49c613a404f855cd11df49b5cc7]) successfully deleted
I20260619 16:50:32.633229 25268 catalog_manager.cc:4841] TS 846e453bc1e8464bbf4be4cc3bf3c037 (127.14.153.3:40983): tablet dd7e5fa3d3bc4918b1764c053a0d3aa4 (table test-table [id=89a1e49c613a404f855cd11df49b5cc7]) successfully deleted
I20260619 16:50:32.635891 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.601944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.14.153.1:36541)
0619 16:50:32.602176 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.602185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.608465 (+ 6280us) server_negotiation.cc:407] Connection header received
0619 16:50:32.608783 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.608792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.608915 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.609001 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:32.611834 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.613303 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.616068 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.616962 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.617057 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.628417 (+ 11360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:32.628435 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:32.628458 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:32.628514 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:32.632616 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.633602 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.633613 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.633616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.633809 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.634297 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.634301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.634303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.634781 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:32.635101 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.635422 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.635580 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:50:32.636551 25468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:32.726040 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 25345
I20260619 16:50:32.743780 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 25479
Jun 19 16:50:32 dist-test-slave-qb36 krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:50:32.759577 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.744683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.14.153.2:38373)
0619 16:50:32.744849 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.744857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.744905 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:50:32.745161 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.745165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.745230 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.745391 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:32.746570 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.747352 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.748613 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.749329 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.749448 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.753804 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:32.753820 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:32.753836 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:32.753873 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:32.756813 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.757511 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.757519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.757521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.757650 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.758016 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.758022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.758026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.758401 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:32.758695 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.759006 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.759247 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:50:32.760257 25602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:32.849970 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 25479
I20260619 16:50:32.867684 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 25613
Jun 19 16:50:32 dist-test-slave-qb36 krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:50:32.886425 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:32.868477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.14.153.3:40983)
0619 16:50:32.868687 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:32.868692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:32.868713 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:50:32.868918 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:32.868921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:32.869006 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:32.869089 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:32.870836 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.871766 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.873707 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:32.874181 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:32.874289 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:32.879939 (+ 5650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:32.879954 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:32.879970 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:32.880017 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:32.883278 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.884174 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.884182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.884184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.884341 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:32.884760 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:32.884763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:32.884765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:32.885103 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:32.885329 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:32.885697 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 16:50:32.886007 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 16:50:32.887285 25736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:33.007068 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 25613
I20260619 16:50:33.026391 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 25252
I20260619 16:50:33.027649 25314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:50:33.134640 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 25252
2026-06-19T16:50:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4256 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:33 dist-test-slave-qb36 krb5kdc[25792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:33 dist-test-slave-qb36 krb5kdc[25792](info): set up 2 sockets
Jun 19 16:50:33 dist-test-slave-qb36 krb5kdc[25792](info): commencing operation
krb5kdc: starting...
W20260619 16:50:34.240123 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:50:34 dist-test-slave-qb36 krb5kdc[25792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/rangeradmin_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/rangeradmin_127.14.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/rangerlookup_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/rangerlookup_127.14.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/HTTP_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/HTTP_127.14.153.20@KRBTEST.COM.keytab.
I20260619 16:50:34.563661 14948 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGGaEbW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/postgres -l logfile start
2026-06-19 16:50:37.105 UTC [25822] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:50:37.105 UTC [25822] LOG: listening on IPv4 address "127.14.153.20", port 39629
2026-06-19 16:50:37.109 UTC [25822] LOG: listening on Unix socket "/tmp/.s.PGSQL.39629"
2026-06-19 16:50:37.132 UTC [25825] LOG: database system was shut down at 2026-06-19 16:50:35 UTC
2026-06-19 16:50:37.136 UTC [25822] LOG: database system is ready to accept connections
I20260619 16:50:38.151748 14948 mini_postgres.cc:96] Postgres bound to 39629
2026-06-19 16:50:38.169 UTC [25833] FATAL: database "slave" does not exist
127.14.153.20:39629 - accepting connections
I20260619 16:50:38.169860 14948 mini_ranger.cc:161] Starting Ranger...
I20260619 16:50:38.196416 14948 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:50:38.705667 14948 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:50:38.705808 14948 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-admin
2026-06-19 16:50:38,985 [I] DB FLAVOR :POSTGRES
2026-06-19 16:50:38,985 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:50:38,985 [I] Checking connection..
2026-06-19 16:50:38,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:50:39,233 [I] Checking connection passed.
2026-06-19 16:50:39,233 [I] --------- Verifying version history table ---------
2026-06-19 16:50:39,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:50:39,498 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:50:39,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:50:39,748 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:50:39,749 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:50:39,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:50:39.986 UTC [25897] WARNING: there is no transaction in progress
2026-06-19 16:50:39,993 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:50:39,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:50:40,228 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:50:40,228 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:50:40,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:50:40,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:50:40,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qb36.c.gcp-upstream.internal','N') ;"
2026-06-19 16:50:40,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:50:41,231 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:50:41,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:50:41,481 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:50:41,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:50:41,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:50:41,966 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:50:41,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:50:42.962 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:42.979 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:42.996 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.012 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.030 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.049 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.107 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.114 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.130 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.140 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.151 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.164 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.175 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.182 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.189 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43.196 UTC [26005] WARNING: there is no transaction in progress
2026-06-19 16:50:43,843 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:50:43,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:50:44,067 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:50:44,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:50:44,315 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:50:44,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:50:44,569 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:50:44,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:50:44,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:50:45,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qb36.c.gcp-upstream.internal';"
2026-06-19 16:50:45,309 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:50:45,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:50:45,555 [I] DB_PATCHES have already been applied
I20260619 16:50:45.561764 14948 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGGaEbW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:50:45.561861 14948 mini_ranger.cc:193] Using host: 127.14.153.20
I20260619 16:50:45.576516 14948 mini_ranger.cc:227] Ranger admin URL: http://127.14.153.20:32897
Jun 19, 2026 4:50:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:50:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:50:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:50:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:50:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:50:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:50:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:50:46.715828 14948 mini_ranger.cc:160] Time spent starting Ranger: real 8.546s user 0.003s sys 0.037s
Jun 19, 2026 4:51:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:51:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 4:51:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:51:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:51:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:51:07.477425 14948 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/rangerkms_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/rangerkms_127.14.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.153.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.153.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.153.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/HTTP_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.153.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/HTTP_127.14.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:51:07 dist-test-slave-qb36 krb5kdc[25792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887867, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:51:07.680219 14948 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:51:07.704589 14948 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:51:08.611299 14948 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:51:08.611474 14948 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-kms
I20260619 16:51:08.611598 14948 mini_ranger_kms.cc:227] Using postgres at 127.14.153.20:39629
2026-06-19 16:51:09,028 [I] DB FLAVOR :POSTGRES
2026-06-19 16:51:09,028 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:51:09,028 [I] Checking connection
2026-06-19 16:51:09,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:51:09,275 [I] connection success
2026-06-19 16:51:09,275 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:51:09,276 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:51:09,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:51:09,505 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:51:09,505 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:51:09,506 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:51:09,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:39629/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:51:09,757 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:51:10.227655 14948 mini_ranger_kms.cc:310] Created kms service
I20260619 16:51:10.420384 14948 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:51:10.472385 14948 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:51:10 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887867, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:51:10.859898 14948 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:51:10.860280 14948 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:51:10.860345 14948 mini_ranger_kms.cc:240] Using host: 127.14.153.20
I20260619 16:51:10.881853 14948 mini_ranger_kms.cc:276] Ranger KMS PID: 26185
I20260619 16:51:10.882086 14948 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.153.20:45431
Jun 19, 2026 4:51:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:51:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 16:51:13 dist-test-slave-qb36 krb5kdc[25792](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887873, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.153.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:13.221957 14948 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.542s user 0.139s sys 0.298s
I20260619 16:51:13.222235 14948 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:51:13.222483 14948 mini_ranger_kms.cc:401] 127.14.153.20:45431/kms/v1/keys
Jun 19 16:51:13 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887873, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.153.20@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
Jun 19 16:51:13 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887873, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.153.20@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
Jun 19 16:51:16 dist-test-slave-qb36 krb5kdc[25792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:51:16.498250 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34105
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34105
--ranger_config_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:45431/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:16.856184 26236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:16.856494 26236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:16.856638 26236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:16.864178 26236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:51:16.864297 26236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:16.864380 26236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:16.864483 26236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:51:16.864548 26236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:51:16.874894 26236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.153.20:45431/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34105
--ranger_config_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:51:16.876350 26236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:16.878243 26236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:16.891991 26241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:16.891950 26242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:16.892299 26244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:16.892891 26236 server_base.cc:1034] running on GCE node
I20260619 16:51:16.893877 26236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:16.895756 26236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:16.896981 26236 hybrid_clock.cc:648] HybridClock initialized: now 1781887876896942 us; error 42 us; skew 500 ppm
I20260619 16:51:16.897398 26236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:51:16 dist-test-slave-qb36 krb5kdc[25792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887876, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:16.904964 26236 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:51:16.906383 26236 webserver.cc:460] Webserver started at http://127.14.153.62:37101/ using document root <none> and password file <none>
I20260619 16:51:16.907014 26236 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:16.907178 26236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:16.907437 26236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:51:16 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887876, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:51:16.955981 26236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "74c5443489994d2b8a8be7dfb3bcf01e"
format_stamp: "Formatted at 2026-06-19 16:51:16 on dist-test-slave-qb36"
server_key: "0adcfd8a827d3fd04a0d2559856b83e1"
server_key_iv: "90cb02ae58a03ea46c31b7f58ec6dd07"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:16.957003 26236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "74c5443489994d2b8a8be7dfb3bcf01e"
format_stamp: "Formatted at 2026-06-19 16:51:16 on dist-test-slave-qb36"
server_key: "0adcfd8a827d3fd04a0d2559856b83e1"
server_key_iv: "90cb02ae58a03ea46c31b7f58ec6dd07"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:16.963151 26236 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260619 16:51:16.990823 26251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:16.992910 26236 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.009s sys 0.001s
I20260619 16:51:16.993119 26236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "74c5443489994d2b8a8be7dfb3bcf01e"
format_stamp: "Formatted at 2026-06-19 16:51:16 on dist-test-slave-qb36"
server_key: "0adcfd8a827d3fd04a0d2559856b83e1"
server_key_iv: "90cb02ae58a03ea46c31b7f58ec6dd07"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:16.993395 26236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:17.021042 26236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:17.025070 26236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:17.025322 26236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:17.045781 26236 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:34105
I20260619 16:51:17.045816 26302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:34105 every 8 connection(s)
I20260619 16:51:17.047212 26236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:51:17.050963 26303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:17.051769 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 26236
I20260619 16:51:17.051972 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:51:17.052788 14948 external_mini_cluster.cc:1428] Setting key 20f6d7a0a85715fa60270f73af41a9cb
I20260619 16:51:17.066104 26303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e: Bootstrap starting.
Jun 19 16:51:17 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:51:17.069671 26303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:17.071264 26303 log.cc:824] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:17.074134 26303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e: No bootstrap required, opened a new log
I20260619 16:51:17.076563 26306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:17.055392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.14.153.62:34105)
0619 16:51:17.056272 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:17.056285 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:17.056334 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:51:17.058045 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:17.058073 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:17.058544 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:17.058899 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:17.060525 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:17.061836 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:17.063695 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:17.064357 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:17.064484 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:17.068884 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:17.068892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:17.068908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:17.068972 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:17.072451 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:17.073149 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:17.073155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:17.073162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:17.073320 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:17.073718 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:17.073722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:17.073725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:17.074280 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:17.074595 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:17.075089 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 16:51:17.075505 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":389,"threads_started":1}
I20260619 16:51:17.082553 26303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5443489994d2b8a8be7dfb3bcf01e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34105 } }
I20260619 16:51:17.083127 26303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:17.083250 26303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c5443489994d2b8a8be7dfb3bcf01e, State: Initialized, Role: FOLLOWER
I20260619 16:51:17.084019 26303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5443489994d2b8a8be7dfb3bcf01e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34105 } }
I20260619 16:51:17.084347 26303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:51:17.084481 26303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:51:17.085052 26303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:17.087282 26303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5443489994d2b8a8be7dfb3bcf01e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34105 } }
I20260619 16:51:17.087738 26303 leader_election.cc:302] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74c5443489994d2b8a8be7dfb3bcf01e; no voters:
I20260619 16:51:17.088322 26303 leader_election.cc:288] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:51:17.088486 26308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:51:17.089159 26308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [term 1 LEADER]: Becoming Leader. State: Replica: 74c5443489994d2b8a8be7dfb3bcf01e, State: Running, Role: LEADER
I20260619 16:51:17.089874 26308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5443489994d2b8a8be7dfb3bcf01e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34105 } }
I20260619 16:51:17.090638 26303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:51:17.092947 26310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c5443489994d2b8a8be7dfb3bcf01e. Latest consensus state: current_term: 1 leader_uuid: "74c5443489994d2b8a8be7dfb3bcf01e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5443489994d2b8a8be7dfb3bcf01e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34105 } } }
I20260619 16:51:17.093418 26310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:17.093333 26309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74c5443489994d2b8a8be7dfb3bcf01e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c5443489994d2b8a8be7dfb3bcf01e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34105 } } }
I20260619 16:51:17.093641 26309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:17.095381 26317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:51:17.100891 26317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:51:17.101854 26303 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:51:17.118969 26317 catalog_manager.cc:1347] Generated new cluster ID: 47a9bdcd3cc242d194ada33dc28d039e
I20260619 16:51:17.119163 26317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:51:17.171969 26317 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:51:17.174131 26317 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:51:17.181845 26317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e: Generated new TSK 0
I20260619 16:51:17.182932 26317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:51:17.941479 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:45431/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:34105
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:51:18 dist-test-slave-qb36 krb5kdc[25792](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:51:18.407783 26348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:18.408210 26348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:18.408416 26348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:18.420316 26348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:18.420475 26348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:18.420710 26348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:51:18.436996 26348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.153.20:45431/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:34105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:51:18.438897 26348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:18.440735 26348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:18.459160 26358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:18.459643 26359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:18.468439 26361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:18.469417 26348 server_base.cc:1034] running on GCE node
I20260619 16:51:18.470389 26348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:18.473256 26348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:18.475157 26348 hybrid_clock.cc:648] HybridClock initialized: now 1781887878475045 us; error 107 us; skew 500 ppm
I20260619 16:51:18.475615 26348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:51:18 dist-test-slave-qb36 krb5kdc[25792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:18.480152 26348 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:51:18.484982 26348 webserver.cc:460] Webserver started at http://127.14.153.1:34739/ using document root <none> and password file <none>
I20260619 16:51:18.486112 26348 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:18.486274 26348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:18.486688 26348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:51:18 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:51:18.514725 26348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "3543cda962f24f8d83485a01eb005d5a"
format_stamp: "Formatted at 2026-06-19 16:51:18 on dist-test-slave-qb36"
server_key: "50cafdb6393ad52d2f78a8db0054ad1a"
server_key_iv: "ffcbf41fc79d0cac6f03812728ca1dd6"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:18.515930 26348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "3543cda962f24f8d83485a01eb005d5a"
format_stamp: "Formatted at 2026-06-19 16:51:18 on dist-test-slave-qb36"
server_key: "50cafdb6393ad52d2f78a8db0054ad1a"
server_key_iv: "ffcbf41fc79d0cac6f03812728ca1dd6"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:18.523334 26348 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260619 16:51:18.543938 26368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:18.545966 26348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:51:18.546160 26348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "3543cda962f24f8d83485a01eb005d5a"
format_stamp: "Formatted at 2026-06-19 16:51:18 on dist-test-slave-qb36"
server_key: "50cafdb6393ad52d2f78a8db0054ad1a"
server_key_iv: "ffcbf41fc79d0cac6f03812728ca1dd6"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:18.546420 26348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:18.598294 26348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:18.604749 26348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:18.605168 26348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:18.618278 26348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:18.626660 26348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:18.626860 26348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 16:51:18.627104 26348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:18.627238 26348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:51:18 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:51:18.955438 26348 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:34627
I20260619 16:51:18.955448 26480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:34627 every 8 connection(s)
I20260619 16:51:18.956962 26348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:51:18.962387 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 26348
I20260619 16:51:18.962635 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:51:18.963802 14948 external_mini_cluster.cc:1428] Setting key 7ae0d79c1310ff07055282f12a7e8730
Jun 19 16:51:19 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:51:19.027482 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:18.979200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:34405 (local address 127.14.153.62:34105)
0619 16:51:18.995130 (+ 15930us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:18.995137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:19.000291 (+ 5154us) server_negotiation.cc:407] Connection header received
0619 16:51:19.003119 (+ 2828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:19.003126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:19.003225 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:19.003303 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:19.005487 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:19.006616 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:19.009012 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:19.009608 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:19.009699 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:19.016086 (+ 6387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:19.016093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:19.016098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:19.016151 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:19.020284 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:19.023160 (+ 2876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:19.023166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:19.023170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:19.023340 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:19.023951 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:19.023956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:19.023959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:19.024308 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:19.024444 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:19.026788 (+ 2344us) server_negotiation.cc:299] Negotiation successful
0619 16:51:19.027111 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15754,"thread_start_us":15610,"threads_started":1}
I20260619 16:51:19.030146 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:18.982294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34105 (local address 127.14.153.1:34405)
0619 16:51:19.000078 (+ 17784us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:19.000143 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:19.002721 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:19.003857 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:19.003869 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:19.004484 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:19.005292 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:19.005314 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:19.006816 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:19.006822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:19.008838 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:19.008853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:19.010052 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:19.010059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:19.010337 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:19.011169 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:19.011198 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:19.015775 (+ 4577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:19.020527 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:19.020537 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:19.020558 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:19.021193 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:19.023513 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:19.023517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:19.023521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:19.023802 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:19.024556 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:19.024564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:19.025087 (+ 523us) client_negotiation.cc:769] Sending connection context
0619 16:51:19.028053 (+ 2966us) client_negotiation.cc:240] Negotiation successful
0619 16:51:19.028649 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17397,"thread_start_us":17109,"threads_started":1}
I20260619 16:51:19.032847 26481 heartbeater.cc:347] Connected to a master server at 127.14.153.62:34105
I20260619 16:51:19.033237 26481 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:19.034312 26481 heartbeater.cc:511] Master 127.14.153.62:34105 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
I20260619 16:51:19.037163 26268 ts_manager.cc:194] Registered new tserver with Master: 3543cda962f24f8d83485a01eb005d5a (127.14.153.1:34627)
I20260619 16:51:19.039563 26268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:34405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
W20260619 16:51:19.069738 14948 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.14.153.2: real 0.102s user 0.002s sys 0.015s
I20260619 16:51:19.070766 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:45431/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:34105
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:51:19 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
W20260619 16:51:19.558084 26487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:19.558485 26487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:19.558681 26487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:19.569597 26487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:19.570243 26487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:19.570531 26487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:51:19.586907 26487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.153.20:45431/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:34105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:51:19.588733 26487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:19.590518 26487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:19.613314 26501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:19.613983 26502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:19.617455 26487 server_base.cc:1034] running on GCE node
W20260619 16:51:19.617697 26504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:19.618471 26487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:19.619787 26487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:19.621177 26487 hybrid_clock.cc:648] HybridClock initialized: now 1781887879621110 us; error 48 us; skew 500 ppm
I20260619 16:51:19.622114 26487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:51:19 dist-test-slave-qb36 krb5kdc[25792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:19.626467 26487 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:51:19.628497 26487 webserver.cc:460] Webserver started at http://127.14.153.2:42145/ using document root <none> and password file <none>
I20260619 16:51:19.629418 26487 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:19.629566 26487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:19.630371 26487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:51:19 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:51:19.652403 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "580523df72774290b8d641cedc5e3e54"
format_stamp: "Formatted at 2026-06-19 16:51:19 on dist-test-slave-qb36"
server_key: "e9eff7b98ad218e5fa57c93ed5827d2e"
server_key_iv: "c325328259f60ff74752f213a97a6680"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:19.653995 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "580523df72774290b8d641cedc5e3e54"
format_stamp: "Formatted at 2026-06-19 16:51:19 on dist-test-slave-qb36"
server_key: "e9eff7b98ad218e5fa57c93ed5827d2e"
server_key_iv: "c325328259f60ff74752f213a97a6680"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:19.677372 26487 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.009s sys 0.009s
I20260619 16:51:19.696728 26511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:19.698480 26487 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:51:19.698674 26487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "580523df72774290b8d641cedc5e3e54"
format_stamp: "Formatted at 2026-06-19 16:51:19 on dist-test-slave-qb36"
server_key: "e9eff7b98ad218e5fa57c93ed5827d2e"
server_key_iv: "c325328259f60ff74752f213a97a6680"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:19.699010 26487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:19.727087 26487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:19.731235 26487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:19.731542 26487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:19.732640 26487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:19.734127 26487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:19.734198 26487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:19.734339 26487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:19.734387 26487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:19.772315 26487 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:38473
I20260619 16:51:19.772330 26623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:38473 every 8 connection(s)
I20260619 16:51:19.773667 26487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:51:19.781960 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 26487
I20260619 16:51:19.782264 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:51:19.783059 14948 external_mini_cluster.cc:1428] Setting key c3c5dd93a0f832cfd07de314ffa85704
Jun 19 16:51:19 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:51:19.800824 26626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:19.776395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:33091 (local address 127.14.153.62:34105)
0619 16:51:19.776847 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:19.776852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:19.777988 (+ 1136us) server_negotiation.cc:407] Connection header received
0619 16:51:19.780080 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:19.780087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:19.780218 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:19.780344 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:19.783162 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:19.784827 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:19.787476 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:19.788094 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:19.788218 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:19.793286 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:19.793296 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:19.793304 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:19.793351 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:19.796638 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:19.797647 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:19.797652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:19.797655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:19.797795 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:19.798343 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:19.798348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:19.798352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:19.798729 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:19.798908 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:19.800213 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0619 16:51:19.800451 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":230,"threads_started":1}
I20260619 16:51:19.801982 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:19.777050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34105 (local address 127.14.153.2:33091)
0619 16:51:19.777817 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:19.777863 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:19.779685 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:19.780680 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:19.780691 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:19.781401 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:19.782252 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:19.782288 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:19.785040 (+ 2752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:19.785045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:19.787285 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:19.787305 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:19.788335 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:19.788346 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:19.788695 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:19.789576 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:19.789610 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:19.793010 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:19.796799 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:19.796811 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:19.796833 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:19.797499 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:19.797924 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:19.797931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:19.797938 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:19.798213 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:19.798909 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:19.798921 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:19.799465 (+ 544us) client_negotiation.cc:769] Sending connection context
0619 16:51:19.799789 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 16:51:19.800437 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":344,"threads_started":1}
I20260619 16:51:19.803774 26624 heartbeater.cc:347] Connected to a master server at 127.14.153.62:34105
I20260619 16:51:19.804111 26624 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:19.805035 26624 heartbeater.cc:511] Master 127.14.153.62:34105 requested a full tablet report, sending...
I20260619 16:51:19.806823 26268 ts_manager.cc:194] Registered new tserver with Master: 580523df72774290b8d641cedc5e3e54 (127.14.153.2:38473)
I20260619 16:51:19.808082 26268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:33091
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:51:19.842718 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:45431/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:34105
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:51:20.044468 26481 heartbeater.cc:503] Master 127.14.153.62:34105 was elected leader, sending a full tablet report...
W20260619 16:51:20.193009 26630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:20.193352 26630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:20.193522 26630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:20.201212 26630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:20.201376 26630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:20.201550 26630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:51:20.212319 26630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.153.20:45431/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:34105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:51:20.213680 26630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:20.215075 26630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:20.228660 26635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:20.228974 26638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:20.229001 26636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:20.230113 26630 server_base.cc:1034] running on GCE node
I20260619 16:51:20.230684 26630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:20.231665 26630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:20.232865 26630 hybrid_clock.cc:648] HybridClock initialized: now 1781887880232825 us; error 38 us; skew 500 ppm
I20260619 16:51:20.233192 26630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:51:20 dist-test-slave-qb36 krb5kdc[25792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887880, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:20.236387 26630 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:51:20.237885 26630 webserver.cc:460] Webserver started at http://127.14.153.3:46161/ using document root <none> and password file <none>
I20260619 16:51:20.238588 26630 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:20.238731 26630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:20.239099 26630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:51:20 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887880, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:51:20.254174 26630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "7ab988e2fb1d4bb49603f88138a1599a"
format_stamp: "Formatted at 2026-06-19 16:51:20 on dist-test-slave-qb36"
server_key: "e58f0e66d1471fd35d3f5f6e810c1f42"
server_key_iv: "3b54c12a4b23457f42c519a54ae708b9"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:20.255005 26630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "7ab988e2fb1d4bb49603f88138a1599a"
format_stamp: "Formatted at 2026-06-19 16:51:20 on dist-test-slave-qb36"
server_key: "e58f0e66d1471fd35d3f5f6e810c1f42"
server_key_iv: "3b54c12a4b23457f42c519a54ae708b9"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:20.260816 26630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:51:20.274998 26645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:20.276890 26630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:51:20.277073 26630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "7ab988e2fb1d4bb49603f88138a1599a"
format_stamp: "Formatted at 2026-06-19 16:51:20 on dist-test-slave-qb36"
server_key: "e58f0e66d1471fd35d3f5f6e810c1f42"
server_key_iv: "3b54c12a4b23457f42c519a54ae708b9"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:20.277304 26630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:20.300596 26630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:20.307670 26630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:20.308387 26630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:20.309376 26630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:20.310782 26630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:20.310885 26630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:20.311074 26630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:20.311154 26630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:51:20.347111 26630 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:41703
I20260619 16:51:20.347147 26757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:41703 every 8 connection(s)
I20260619 16:51:20.348416 26630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:51:20.348978 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 26630
I20260619 16:51:20.349159 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:51:20.349728 14948 external_mini_cluster.cc:1428] Setting key cfa5244cfb6d35f977157544ab263568
Jun 19 16:51:20 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887880, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:51:20.370123 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.351246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:44037 (local address 127.14.153.62:34105)
0619 16:51:20.351833 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.351839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.352925 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 16:51:20.355286 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.355292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.355393 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.355512 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:20.356973 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.357685 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.359322 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.359767 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.359903 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.363374 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:20.363380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:20.363383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:20.363416 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:20.366524 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:20.367474 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:20.367478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:20.367480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:20.367613 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:20.368219 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:20.368228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:20.368233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:20.368693 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:20.368886 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.369688 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.369863 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":304,"threads_started":1}
I20260619 16:51:20.371294 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.351848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34105 (local address 127.14.153.3:44037)
0619 16:51:20.352715 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:20.352794 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:20.354955 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:20.355761 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:20.355770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:20.356195 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:20.356801 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.356818 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.357837 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.357842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.359176 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.359188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.360016 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.360022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.360255 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.360984 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:20.361023 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:20.363128 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:20.366683 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:20.366690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:20.366705 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:20.367259 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:20.367722 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:20.367725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:20.367727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:20.367982 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:20.368861 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:20.368871 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:20.369331 (+ 460us) client_negotiation.cc:769] Sending connection context
0619 16:51:20.369590 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 16:51:20.369995 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":354,"threads_started":1}
I20260619 16:51:20.372815 26758 heartbeater.cc:347] Connected to a master server at 127.14.153.62:34105
I20260619 16:51:20.373117 26758 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:20.373780 26758 heartbeater.cc:511] Master 127.14.153.62:34105 requested a full tablet report, sending...
I20260619 16:51:20.375231 26268 ts_manager.cc:194] Registered new tserver with Master: 7ab988e2fb1d4bb49603f88138a1599a (127.14.153.3:41703)
I20260619 16:51:20.376099 26268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:44037
I20260619 16:51:20.378271 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:51:20.396358 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.383436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.14.153.62:34105)
0619 16:51:20.383657 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.383666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.384111 (+ 445us) server_negotiation.cc:407] Connection header received
0619 16:51:20.384239 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.384243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.384320 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.384413 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:20.385846 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.386601 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.388102 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.388536 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.388637 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.391163 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:20.391170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:20.391173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:20.391207 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:20.393687 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:20.394396 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:20.394400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:20.394402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:20.394526 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:20.394927 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:20.394930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:20.394932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:20.395304 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:20.395489 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.395800 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.396046 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:51:20.400795 26268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:51:20.409030 26268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:51:20.444749 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.421959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.14.153.2:38473)
0619 16:51:20.433781 (+ 11822us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.433796 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.434834 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 16:51:20.435319 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.435328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.435602 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.435769 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:20.437174 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.439331 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.441846 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.443406 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.443597 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.443806 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.444198 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.444365 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11553,"thread_start_us":11828,"threads_started":1}
I20260619 16:51:20.444778 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.421878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:38473 (local address 127.0.0.1:47278)
0619 16:51:20.434659 (+ 12781us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:20.434699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:20.434995 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:20.436153 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:20.436164 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:20.436208 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:20.436880 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.436897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.439644 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.439653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.441678 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.441691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.443665 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.443671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.443882 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.443972 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 16:51:20.444128 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 16:51:20.444352 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12566,"mutex_wait_us":2,"thread_start_us":12414,"threads_started":1}
I20260619 16:51:20.445407 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.421844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:34627 (local address 127.0.0.1:43152)
0619 16:51:20.434856 (+ 13012us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:20.434874 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:20.435014 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:20.436275 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:20.436281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:20.436329 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:20.436880 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.436898 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.439282 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.439288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.442609 (+ 3321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.442621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.444565 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.444572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.444747 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.444794 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 16:51:20.444894 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:51:20.445035 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12510,"mutex_wait_us":97,"spinlock_wait_cycles":8576,"thread_start_us":12565,"threads_started":1}
I20260619 16:51:20.446429 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.422312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.14.153.1:34627)
0619 16:51:20.434389 (+ 12077us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.434398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.435255 (+ 857us) server_negotiation.cc:407] Connection header received
0619 16:51:20.435363 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.435370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.436063 (+ 693us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.437151 (+ 1088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:20.437447 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.439132 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.442789 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.444327 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.445756 (+ 1429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.445932 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.445988 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.446106 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11880,"thread_start_us":12310,"threads_started":1}
I20260619 16:51:20.449525 26559 tablet_service.cc:1467] Processing CreateTablet for tablet e28e1ba782e44bef8a2bf95f1d48b508 (DEFAULT_TABLE table=test-table [id=84ef8a6cd8b34d91ba114d4d0d64ea00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:20.449934 26416 tablet_service.cc:1467] Processing CreateTablet for tablet e28e1ba782e44bef8a2bf95f1d48b508 (DEFAULT_TABLE table=test-table [id=84ef8a6cd8b34d91ba114d4d0d64ea00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:20.451011 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.436519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:41703 (local address 127.0.0.1:43442)
0619 16:51:20.442234 (+ 5715us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:20.442257 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:20.442384 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:20.443329 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:20.443334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:20.443359 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:20.443780 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.443791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.445706 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.445710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.448299 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.448313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.450158 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.450166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.450343 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.450375 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:51:20.450477 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:51:20.450632 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5581,"thread_start_us":6608,"threads_started":1}
I20260619 16:51:20.451093 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.435616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.14.153.3:41703)
0619 16:51:20.440151 (+ 4535us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.440161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.442508 (+ 2347us) server_negotiation.cc:407] Connection header received
0619 16:51:20.442620 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.442629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.442918 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.443158 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:20.443986 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.445535 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.448482 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.449917 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.450069 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.450292 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.450541 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.450717 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4347,"thread_start_us":4110,"threads_started":1}
I20260619 16:51:20.451102 26559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28e1ba782e44bef8a2bf95f1d48b508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:20.451486 26416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28e1ba782e44bef8a2bf95f1d48b508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:20.455133 26693 tablet_service.cc:1467] Processing CreateTablet for tablet e28e1ba782e44bef8a2bf95f1d48b508 (DEFAULT_TABLE table=test-table [id=84ef8a6cd8b34d91ba114d4d0d64ea00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:20.456800 26693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28e1ba782e44bef8a2bf95f1d48b508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:20.469290 26777 tablet_bootstrap.cc:492] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: Bootstrap starting.
I20260619 16:51:20.471098 26778 tablet_bootstrap.cc:492] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: Bootstrap starting.
I20260619 16:51:20.473318 26777 tablet_bootstrap.cc:654] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:20.474709 26778 tablet_bootstrap.cc:654] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:20.476485 26777 log.cc:824] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:20.476622 26778 log.cc:824] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:20.476800 26779 tablet_bootstrap.cc:492] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: Bootstrap starting.
I20260619 16:51:20.482030 26779 tablet_bootstrap.cc:654] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:20.484388 26779 log.cc:824] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:20.488898 26779 tablet_bootstrap.cc:492] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: No bootstrap required, opened a new log
I20260619 16:51:20.488898 26778 tablet_bootstrap.cc:492] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: No bootstrap required, opened a new log
I20260619 16:51:20.489198 26777 tablet_bootstrap.cc:492] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: No bootstrap required, opened a new log
I20260619 16:51:20.489209 26778 ts_tablet_manager.cc:1397] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.010s
I20260619 16:51:20.489207 26779 ts_tablet_manager.cc:1397] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260619 16:51:20.489492 26777 ts_tablet_manager.cc:1397] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.010s
I20260619 16:51:20.500479 26779 raft_consensus.cc:348] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } }
I20260619 16:51:20.500479 26778 raft_consensus.cc:348] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } }
I20260619 16:51:20.500985 26779 raft_consensus.cc:374] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:20.500988 26778 raft_consensus.cc:374] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:20.501093 26778 raft_consensus.cc:729] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3543cda962f24f8d83485a01eb005d5a, State: Initialized, Role: FOLLOWER
I20260619 16:51:20.501104 26779 raft_consensus.cc:729] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab988e2fb1d4bb49603f88138a1599a, State: Initialized, Role: FOLLOWER
I20260619 16:51:20.501927 26779 consensus_queue.cc:260] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } }
I20260619 16:51:20.501929 26778 consensus_queue.cc:260] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } }
I20260619 16:51:20.503705 26758 heartbeater.cc:503] Master 127.14.153.62:34105 was elected leader, sending a full tablet report...
I20260619 16:51:20.504487 26779 ts_tablet_manager.cc:1428] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260619 16:51:20.505745 26778 ts_tablet_manager.cc:1428] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260619 16:51:20.506771 26777 raft_consensus.cc:348] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } }
I20260619 16:51:20.507361 26777 raft_consensus.cc:374] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:20.507483 26777 raft_consensus.cc:729] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580523df72774290b8d641cedc5e3e54, State: Initialized, Role: FOLLOWER
I20260619 16:51:20.508148 26777 consensus_queue.cc:260] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } }
I20260619 16:51:20.509997 26624 heartbeater.cc:503] Master 127.14.153.62:34105 was elected leader, sending a full tablet report...
I20260619 16:51:20.509999 26777 ts_tablet_manager.cc:1428] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: Time spent starting tablet: real 0.020s user 0.008s sys 0.008s
W20260619 16:51:20.525712 26625 tablet.cc:2369] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:51:20.600206 26759 tablet.cc:2369] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:51:20.646394 26784 raft_consensus.cc:482] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:51:20.646734 26784 raft_consensus.cc:504] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } }
I20260619 16:51:20.648545 26784 leader_election.cc:288] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 580523df72774290b8d641cedc5e3e54 (127.14.153.2:38473), 7ab988e2fb1d4bb49603f88138a1599a (127.14.153.3:41703)
I20260619 16:51:20.655787 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.649807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:37403 (local address 127.14.153.3:41703)
0619 16:51:20.650038 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.650044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.650397 (+ 353us) server_negotiation.cc:407] Connection header received
0619 16:51:20.650564 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.650568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.650624 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.650731 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:20.651275 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.652067 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.654026 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.654811 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.655008 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.655109 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.655400 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.655523 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:51:20.655903 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.649672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:41703 (local address 127.14.153.1:37403)
0619 16:51:20.650282 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:20.650300 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:20.650435 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:20.650761 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:20.650763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:20.650791 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:20.651131 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.651140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.652244 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.652265 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.653877 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.653892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.655058 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.655064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.655242 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.655265 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:51:20.655378 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:51:20.655513 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":276,"threads_started":1}
I20260619 16:51:20.656304 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.648946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:47251 (local address 127.14.153.2:38473)
0619 16:51:20.649154 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.649163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.649634 (+ 471us) server_negotiation.cc:407] Connection header received
0619 16:51:20.649897 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.649907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.650003 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.650102 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:20.650726 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.651691 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.653774 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.654846 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.655011 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.655109 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.655566 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.655690 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:51:20.656323 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.648926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:38473 (local address 127.14.153.1:47251)
0619 16:51:20.649525 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:20.649543 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:20.649716 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:20.650143 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:20.650148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:20.650173 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:20.650585 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.650598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.651876 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.651882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.653609 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:20.653623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.655260 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.655265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:20.655406 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.655428 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:51:20.655541 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:51:20.655669 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":102,"thread_start_us":300,"threads_started":1}
I20260619 16:51:20.656874 26713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28e1ba782e44bef8a2bf95f1d48b508" candidate_uuid: "3543cda962f24f8d83485a01eb005d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab988e2fb1d4bb49603f88138a1599a" is_pre_election: true
I20260619 16:51:20.657156 26579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28e1ba782e44bef8a2bf95f1d48b508" candidate_uuid: "3543cda962f24f8d83485a01eb005d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580523df72774290b8d641cedc5e3e54" is_pre_election: true
I20260619 16:51:20.657398 26713 raft_consensus.cc:2454] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3543cda962f24f8d83485a01eb005d5a in term 0.
I20260619 16:51:20.657661 26579 raft_consensus.cc:2454] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3543cda962f24f8d83485a01eb005d5a in term 0.
I20260619 16:51:20.658082 26371 leader_election.cc:302] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3543cda962f24f8d83485a01eb005d5a, 7ab988e2fb1d4bb49603f88138a1599a; no voters:
I20260619 16:51:20.658468 26784 raft_consensus.cc:2781] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:51:20.658610 26784 raft_consensus.cc:482] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:51:20.658717 26784 raft_consensus.cc:3037] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:20.661305 26784 raft_consensus.cc:504] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } }
I20260619 16:51:20.661896 26784 leader_election.cc:288] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [CANDIDATE]: Term 1 election: Requested vote from peers 580523df72774290b8d641cedc5e3e54 (127.14.153.2:38473), 7ab988e2fb1d4bb49603f88138a1599a (127.14.153.3:41703)
I20260619 16:51:20.662298 26579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28e1ba782e44bef8a2bf95f1d48b508" candidate_uuid: "3543cda962f24f8d83485a01eb005d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580523df72774290b8d641cedc5e3e54"
I20260619 16:51:20.662374 26713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28e1ba782e44bef8a2bf95f1d48b508" candidate_uuid: "3543cda962f24f8d83485a01eb005d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab988e2fb1d4bb49603f88138a1599a"
I20260619 16:51:20.662500 26579 raft_consensus.cc:3037] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:20.662612 26713 raft_consensus.cc:3037] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:20.664767 26579 raft_consensus.cc:2454] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3543cda962f24f8d83485a01eb005d5a in term 1.
I20260619 16:51:20.664783 26713 raft_consensus.cc:2454] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3543cda962f24f8d83485a01eb005d5a in term 1.
I20260619 16:51:20.665192 26371 leader_election.cc:302] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3543cda962f24f8d83485a01eb005d5a, 7ab988e2fb1d4bb49603f88138a1599a; no voters:
I20260619 16:51:20.665516 26784 raft_consensus.cc:2781] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:51:20.665886 26784 raft_consensus.cc:686] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 1 LEADER]: Becoming Leader. State: Replica: 3543cda962f24f8d83485a01eb005d5a, State: Running, Role: LEADER
I20260619 16:51:20.666342 26784 consensus_queue.cc:237] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } }
I20260619 16:51:20.670756 26267 catalog_manager.cc:5495] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3543cda962f24f8d83485a01eb005d5a (127.14.153.1). New cstate: current_term: 1 leader_uuid: "3543cda962f24f8d83485a01eb005d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3543cda962f24f8d83485a01eb005d5a" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 34627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 } health_report { overall_health: UNKNOWN } } }
W20260619 16:51:20.716217 26482 tablet.cc:2369] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:51:20.743687 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.738377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.14.153.1:34627)
0619 16:51:20.738560 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.738564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.738581 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:51:20.738721 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.738725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.738807 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.738964 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:51:20.739421 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.740159 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.741666 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.742058 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.742158 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.742574 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:51:20.743215 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:51:20.743328 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.743415 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.743484 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:51:20.751936 26713 raft_consensus.cc:1261] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [term 1 FOLLOWER]: Refusing update from remote peer 3543cda962f24f8d83485a01eb005d5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:51:20.752019 26579 raft_consensus.cc:1261] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [term 1 FOLLOWER]: Refusing update from remote peer 3543cda962f24f8d83485a01eb005d5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:51:20.752995 26788 consensus_queue.cc:1035] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab988e2fb1d4bb49603f88138a1599a" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 41703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:51:20.753387 26784 consensus_queue.cc:1035] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "580523df72774290b8d641cedc5e3e54" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:51:20.765249 26791 mvcc.cc:205] Tried to move back new op lower bound from 7298612759550091264 to 7298612759213199360. Current Snapshot: MvccSnapshot[applied={T|T < 7298612759550091264}]
I20260619 16:51:20.779690 26267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:51:20.780023 26267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:51:20.784646 26267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74c5443489994d2b8a8be7dfb3bcf01e: Sending DeleteTablet for 3 replicas of tablet e28e1ba782e44bef8a2bf95f1d48b508
I20260619 16:51:20.785775 26559 tablet_service.cc:1514] Processing DeleteTablet for tablet e28e1ba782e44bef8a2bf95f1d48b508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:20 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:47278
I20260619 16:51:20.785792 26416 tablet_service.cc:1514] Processing DeleteTablet for tablet e28e1ba782e44bef8a2bf95f1d48b508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:20 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:43152
I20260619 16:51:20.786717 26800 tablet_replica.cc:330] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: stopping tablet replica
I20260619 16:51:20.787231 26800 raft_consensus.cc:2229] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:51:20.787819 26800 raft_consensus.cc:2258] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:51:20.788019 26693 tablet_service.cc:1514] Processing DeleteTablet for tablet e28e1ba782e44bef8a2bf95f1d48b508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:20 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:43442
I20260619 16:51:20.789947 26800 ts_tablet_manager.cc:1905] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:51:20.792538 26802 tablet_replica.cc:330] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: stopping tablet replica
I20260619 16:51:20.792986 26802 raft_consensus.cc:2229] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:51:20.793512 26802 raft_consensus.cc:2258] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:51:20.794648 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 26348
I20260619 16:51:20.797730 26800 ts_tablet_manager.cc:1918] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:51:20.797979 26800 log.cc:1196] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/e28e1ba782e44bef8a2bf95f1d48b508
I20260619 16:51:20.798588 26800 ts_tablet_manager.cc:1939] T e28e1ba782e44bef8a2bf95f1d48b508 P 580523df72774290b8d641cedc5e3e54: Deleting consensus metadata
I20260619 16:51:20.800478 26254 catalog_manager.cc:4841] TS 580523df72774290b8d641cedc5e3e54 (127.14.153.2:38473): tablet e28e1ba782e44bef8a2bf95f1d48b508 (table test-table [id=84ef8a6cd8b34d91ba114d4d0d64ea00]) successfully deleted
I20260619 16:51:20.800822 26802 ts_tablet_manager.cc:1905] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:51:20.809310 26802 ts_tablet_manager.cc:1918] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:51:20.809520 26802 log.cc:1196] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/e28e1ba782e44bef8a2bf95f1d48b508
I20260619 16:51:20.809995 26802 ts_tablet_manager.cc:1939] T e28e1ba782e44bef8a2bf95f1d48b508 P 7ab988e2fb1d4bb49603f88138a1599a: Deleting consensus metadata
I20260619 16:51:20.810927 26801 tablet_replica.cc:330] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: stopping tablet replica
I20260619 16:51:20.811470 26801 raft_consensus.cc:2229] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:51:20.812041 26254 catalog_manager.cc:4841] TS 7ab988e2fb1d4bb49603f88138a1599a (127.14.153.3:41703): tablet e28e1ba782e44bef8a2bf95f1d48b508 (table test-table [id=84ef8a6cd8b34d91ba114d4d0d64ea00]) successfully deleted
I20260619 16:51:20.812403 26801 raft_consensus.cc:2258] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:51:20.814525 26801 ts_tablet_manager.cc:1905] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:51:20 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:51:20.822328 26801 ts_tablet_manager.cc:1918] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:51:20.822521 26801 log.cc:1196] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/e28e1ba782e44bef8a2bf95f1d48b508
I20260619 16:51:20.823169 26801 ts_tablet_manager.cc:1939] T e28e1ba782e44bef8a2bf95f1d48b508 P 3543cda962f24f8d83485a01eb005d5a: Deleting consensus metadata
I20260619 16:51:20.825549 26255 catalog_manager.cc:4841] TS 3543cda962f24f8d83485a01eb005d5a (127.14.153.1:34627): tablet e28e1ba782e44bef8a2bf95f1d48b508 (table test-table [id=84ef8a6cd8b34d91ba114d4d0d64ea00]) successfully deleted
I20260619 16:51:20.827358 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.795760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.14.153.1:34627)
0619 16:51:20.795948 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.795956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.810137 (+ 14181us) server_negotiation.cc:407] Connection header received
0619 16:51:20.810405 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.810411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.810538 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.810668 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:20.812594 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.813776 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.815201 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.815660 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.815822 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.820274 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:20.820287 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:20.820301 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:20.820337 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:20.824314 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:20.825059 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:20.825067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:20.825069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:20.825218 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:20.825722 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:20.825727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:20.825730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:20.826107 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:20.826432 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.826738 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.826967 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:51:20.827970 26476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:51:20.929739 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 26348
I20260619 16:51:20.948323 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 26487
Jun 19 16:51:20 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:51:20.963752 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:20.949172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.14.153.2:38473)
0619 16:51:20.949348 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:20.949352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:20.949373 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:51:20.949430 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:20.949435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:20.949492 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:20.949573 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:20.950935 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.951765 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.953071 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:20.953518 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:20.953639 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:20.958020 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:20.958037 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:20.958054 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:20.958090 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:20.960976 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:20.961749 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:20.961757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:20.961759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:20.961887 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:20.962248 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:20.962251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:20.962253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:20.962615 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:20.962820 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:20.963305 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 16:51:20.963492 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:51:20.964507 26619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:51:21.057447 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 26487
I20260619 16:51:21.077917 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 26630
Jun 19 16:51:21 dist-test-slave-qb36 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:51:21.093549 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:21.078697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.14.153.3:41703)
0619 16:51:21.078886 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:21.078891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:21.078909 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:51:21.079149 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:21.079153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:21.079218 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:21.079306 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:21.080780 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:21.081489 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:21.083108 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:21.083502 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:21.083596 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:21.087984 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:21.088001 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:21.088019 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:21.088059 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:21.090699 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:21.091627 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:21.091635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:21.091637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:21.091777 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:21.092178 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:21.092181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:21.092183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:21.092544 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:21.092748 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:21.093021 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 16:51:21.093178 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:51:21.094161 26753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:51:21.183661 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 26630
I20260619 16:51:21.200524 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 26236
I20260619 16:51:21.201990 26298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:51:21.315506 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 26236
I20260619 16:51:21.338389 14948 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:51:21.674252 14948 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:51:21.674531 14948 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:51:22.020534 14948 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:51:22.020 UTC [25822] LOG: received smart shutdown request
2026-06-19 16:51:22.024 UTC [25822] LOG: background worker "logical replication launcher" (PID 25831) exited with exit code 1
2026-06-19 16:51:22.024 UTC [25826] LOG: shutting down
2026-06-19 16:51:22.205 UTC [25822] LOG: database system is shut down
2026-06-19T16:51:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49203 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:51:22 dist-test-slave-qb36 krb5kdc[26816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:51:22 dist-test-slave-qb36 krb5kdc[26816](info): set up 2 sockets
Jun 19 16:51:22 dist-test-slave-qb36 krb5kdc[26816](info): commencing operation
krb5kdc: starting...
W20260619 16:51:23.447526 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:51:23 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:51:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/rangeradmin_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/rangeradmin_127.14.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/rangerlookup_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/rangerlookup_127.14.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/HTTP_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/HTTP_127.14.153.20@KRBTEST.COM.keytab.
I20260619 16:51:23.767674 14948 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGGaEbW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/postgres -l logfile start
2026-06-19 16:51:26.318 UTC [26846] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:51:26.318 UTC [26846] LOG: listening on IPv4 address "127.14.153.20", port 58211
2026-06-19 16:51:26.322 UTC [26846] LOG: listening on Unix socket "/tmp/.s.PGSQL.58211"
2026-06-19 16:51:26.343 UTC [26849] LOG: database system was shut down at 2026-06-19 16:51:24 UTC
2026-06-19 16:51:26.347 UTC [26846] LOG: database system is ready to accept connections
I20260619 16:51:27.323104 14948 mini_postgres.cc:96] Postgres bound to 58211
2026-06-19 16:51:27.340 UTC [26857] FATAL: database "slave" does not exist
127.14.153.20:58211 - accepting connections
I20260619 16:51:27.340765 14948 mini_ranger.cc:161] Starting Ranger...
I20260619 16:51:27.367865 14948 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:51:27.867093 14948 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:51:27.867194 14948 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-admin
2026-06-19 16:51:28,142 [I] DB FLAVOR :POSTGRES
2026-06-19 16:51:28,143 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:51:28,143 [I] Checking connection..
2026-06-19 16:51:28,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:51:28,378 [I] Checking connection passed.
2026-06-19 16:51:28,379 [I] --------- Verifying version history table ---------
2026-06-19 16:51:28,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:51:28,614 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:51:28,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:51:28,878 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:51:28,878 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:51:28,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:51:29.132 UTC [26921] WARNING: there is no transaction in progress
2026-06-19 16:51:29,138 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:51:29,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:51:29,375 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:51:29,376 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:51:29,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:51:29,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:51:29,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qb36.c.gcp-upstream.internal','N') ;"
2026-06-19 16:51:30,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:51:30,355 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:51:30,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:51:30,607 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:51:30,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:51:30,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:51:31,109 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:51:31,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:51:32.101 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.119 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.137 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.153 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.170 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.189 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.246 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.253 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.267 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.278 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.289 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.303 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.313 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.320 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.328 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32.335 UTC [27029] WARNING: there is no transaction in progress
2026-06-19 16:51:32,976 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:51:32,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:51:33,211 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:51:33,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:51:33,474 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:51:33,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:51:33,738 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:51:33,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:51:33,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:51:34,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qb36.c.gcp-upstream.internal';"
2026-06-19 16:51:34,509 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:51:34,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:51:34,767 [I] DB_PATCHES have already been applied
I20260619 16:51:34.773649 14948 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGGaEbW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:51:34.773742 14948 mini_ranger.cc:193] Using host: 127.14.153.20
I20260619 16:51:34.788473 14948 mini_ranger.cc:227] Ranger admin URL: http://127.14.153.20:52335
Jun 19, 2026 4:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:51:35.917598 14948 mini_ranger.cc:160] Time spent starting Ranger: real 8.577s user 0.005s sys 0.034s
Jun 19, 2026 4:51:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:51:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Jun 19, 2026 4:51:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:51:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:51:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:51:57.156154 14948 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/rangerkms_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/rangerkms_127.14.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.153.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.153.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.153.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/HTTP_127.14.153.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.153.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/HTTP_127.14.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:51:57 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887917, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:51:57.323604 14948 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:51:57.344801 14948 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:51:58.240725 14948 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:51:58.240845 14948 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-kms
I20260619 16:51:58.240890 14948 mini_ranger_kms.cc:227] Using postgres at 127.14.153.20:58211
2026-06-19 16:51:58,597 [I] DB FLAVOR :POSTGRES
2026-06-19 16:51:58,597 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:51:58,597 [I] Checking connection
2026-06-19 16:51:58,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:51:58,828 [I] connection success
2026-06-19 16:51:58,829 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:51:58,829 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:51:58,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:51:59,093 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:51:59,093 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:51:59,093 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:51:59,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.153.20:58211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:51:59,366 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:51:59.852968 14948 mini_ranger_kms.cc:310] Created kms service
I20260619 16:52:00.057302 14948 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:52:00.102519 14948 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:52:00 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887917, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:00.517287 14948 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:52:00.517630 14948 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGGaEbW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGGaEbW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:52:00.517679 14948 mini_ranger_kms.cc:240] Using host: 127.14.153.20
I20260619 16:52:00.534842 14948 mini_ranger_kms.cc:276] Ranger KMS PID: 27209
I20260619 16:52:00.534979 14948 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.153.20:47457
Jun 19, 2026 4:52:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:52:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:52:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:52:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:52:02.859258 14948 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.536s user 0.083s sys 0.292s
I20260619 16:52:02.859536 14948 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:52:02.859740 14948 mini_ranger_kms.cc:401] 127.14.153.20:47457/kms/v1/keys
Jun 19 16:52:02 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887922, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.153.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:52:03 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887922, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.153.20@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
Jun 19 16:52:03 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887922, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.153.20@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
Jun 19 16:52:06 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:52:06.152999 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35007
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35007
--ranger_config_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:47457/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:06.510071 27260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:06.510434 27260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:06.510589 27260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:06.517932 27260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:06.518103 27260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:06.518205 27260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:06.518306 27260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:06.518395 27260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:06.528887 27260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.153.20:47457/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35007
--ranger_config_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:06.530259 27260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:06.531729 27260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:06.544477 27265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:06.544446 27268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:06.544384 27266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:06.545130 27260 server_base.cc:1034] running on GCE node
I20260619 16:52:06.545923 27260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:06.547410 27260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:06.548593 27260 hybrid_clock.cc:648] HybridClock initialized: now 1781887926548561 us; error 32 us; skew 500 ppm
I20260619 16:52:06.548977 27260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:06 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887926, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:06.565274 27260 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:52:06.567013 27260 webserver.cc:460] Webserver started at http://127.14.153.62:45805/ using document root <none> and password file <none>
I20260619 16:52:06.567786 27260 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:06.567929 27260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:06.568244 27260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:52:06 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887926, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:06.627699 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "6fef7229efb6427899bcfbfdca5f3578"
format_stamp: "Formatted at 2026-06-19 16:52:06 on dist-test-slave-qb36"
server_key: "2e33fadeb62c36be4376df0fb9dd1f7b"
server_key_iv: "93ddf0639dc6b9aedd4c61460727c813"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:06.628635 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "6fef7229efb6427899bcfbfdca5f3578"
format_stamp: "Formatted at 2026-06-19 16:52:06 on dist-test-slave-qb36"
server_key: "2e33fadeb62c36be4376df0fb9dd1f7b"
server_key_iv: "93ddf0639dc6b9aedd4c61460727c813"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:06.634572 27260 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 16:52:06.652590 27275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:06.654184 27260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:52:06.654426 27260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "6fef7229efb6427899bcfbfdca5f3578"
format_stamp: "Formatted at 2026-06-19 16:52:06 on dist-test-slave-qb36"
server_key: "2e33fadeb62c36be4376df0fb9dd1f7b"
server_key_iv: "93ddf0639dc6b9aedd4c61460727c813"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:06.654681 27260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:06.671788 27260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:06.676252 27260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:06.676602 27260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:06.695999 27260 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35007
I20260619 16:52:06.696022 27326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35007 every 8 connection(s)
I20260619 16:52:06.697479 27260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:06.701547 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 27260
I20260619 16:52:06.701632 27327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:06.701792 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:52:06.702500 14948 external_mini_cluster.cc:1428] Setting key 0419d0f49c061c94695cf52593f73551
I20260619 16:52:06.715695 27327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Bootstrap starting.
Jun 19 16:52:06 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:06.719668 27327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:06.720930 27327 log.cc:824] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:06.724047 27327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: No bootstrap required, opened a new log
I20260619 16:52:06.726786 27330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:06.704587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.14.153.62:35007)
0619 16:52:06.705387 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:06.705399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:06.705445 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:52:06.707710 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:06.707735 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:06.708193 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:06.708553 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:06.710226 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:06.711640 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:06.713532 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:06.714264 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:06.714406 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:06.719174 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:06.719184 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:06.719201 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:06.719242 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:06.722452 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:06.723275 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:06.723282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:06.723289 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:06.723455 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:06.723888 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:06.723899 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:06.723902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:06.724500 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:06.724832 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:06.725264 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 16:52:06.725710 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":383,"threads_started":1}
I20260619 16:52:06.732080 27327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } }
I20260619 16:52:06.732462 27327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:06.732573 27327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fef7229efb6427899bcfbfdca5f3578, State: Initialized, Role: FOLLOWER
I20260619 16:52:06.733237 27327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } }
I20260619 16:52:06.733481 27327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:06.733630 27327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:06.733811 27327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:06.736474 27327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } }
I20260619 16:52:06.737012 27327 leader_election.cc:302] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fef7229efb6427899bcfbfdca5f3578; no voters:
I20260619 16:52:06.737778 27327 leader_election.cc:288] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:52:06.737919 27332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:06.738693 27332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 1 LEADER]: Becoming Leader. State: Replica: 6fef7229efb6427899bcfbfdca5f3578, State: Running, Role: LEADER
I20260619 16:52:06.739287 27332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } }
I20260619 16:52:06.740036 27327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:06.741962 27334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fef7229efb6427899bcfbfdca5f3578. Latest consensus state: current_term: 1 leader_uuid: "6fef7229efb6427899bcfbfdca5f3578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } } }
I20260619 16:52:06.742352 27334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:06.742866 27333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fef7229efb6427899bcfbfdca5f3578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } } }
I20260619 16:52:06.743292 27333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:06.743455 27338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:06.749758 27338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:06.750576 27327 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:52:06.767467 27338 catalog_manager.cc:1347] Generated new cluster ID: f97ec72fc422435ebee235290323d8bd
I20260619 16:52:06.767630 27338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:06.783993 27338 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:52:06.785590 27338 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:06.791648 27338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Generated new TSK 0
I20260619 16:52:06.792562 27338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:52:07.631465 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:47457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35007
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:52:07 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887927, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:52:08.051432 27372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:08.051759 27372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:08.051937 27372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:08.060235 27372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:08.060411 27372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:08.060608 27372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:52:08.071017 27372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.153.20:47457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:08.072497 27372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:08.073935 27372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:08.091082 27383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:08.091565 27382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:08.092074 27385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:08.093593 27372 server_base.cc:1034] running on GCE node
I20260619 16:52:08.094411 27372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:08.095774 27372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:08.099144 27372 hybrid_clock.cc:648] HybridClock initialized: now 1781887928099022 us; error 101 us; skew 500 ppm
I20260619 16:52:08.099678 27372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:08 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887928, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:08.103894 27372 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:52:08.112689 27372 webserver.cc:460] Webserver started at http://127.14.153.1:41771/ using document root <none> and password file <none>
I20260619 16:52:08.113722 27372 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:08.113870 27372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:08.114228 27372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:52:08 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887928, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:08.146456 27372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "945f1a3d1c2044718cda0c80fe639357"
format_stamp: "Formatted at 2026-06-19 16:52:08 on dist-test-slave-qb36"
server_key: "9541d397bad264402954e5fdf63d6e30"
server_key_iv: "7dde660e93f9486573b5392db88d025f"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:08.147913 27372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "945f1a3d1c2044718cda0c80fe639357"
format_stamp: "Formatted at 2026-06-19 16:52:08 on dist-test-slave-qb36"
server_key: "9541d397bad264402954e5fdf63d6e30"
server_key_iv: "7dde660e93f9486573b5392db88d025f"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:08.167466 27372 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.004s sys 0.011s
I20260619 16:52:08.193351 27392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:08.195631 27372 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.006s
I20260619 16:52:08.195868 27372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "945f1a3d1c2044718cda0c80fe639357"
format_stamp: "Formatted at 2026-06-19 16:52:08 on dist-test-slave-qb36"
server_key: "9541d397bad264402954e5fdf63d6e30"
server_key_iv: "7dde660e93f9486573b5392db88d025f"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:08.196138 27372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:08.227499 27372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:08.233012 27372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:08.233420 27372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:08.237676 27372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:08.239704 27372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:08.239843 27372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 16:52:08.240000 27372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:08.240083 27372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:08.301461 27372 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:45191
I20260619 16:52:08.303361 27372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:52:08.303308 27504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:45191 every 8 connection(s)
I20260619 16:52:08.313079 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 27372
I20260619 16:52:08.313428 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:52:08.314595 14948 external_mini_cluster.cc:1428] Setting key bf6bf9bd90f84e6a037ecfd7dc17441a
Jun 19 16:52:08 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887928, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:52:08.372224 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:08.307641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:35759 (local address 127.14.153.62:35007)
0619 16:52:08.329094 (+ 21453us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:08.329103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:08.341405 (+ 12302us) server_negotiation.cc:407] Connection header received
0619 16:52:08.351177 (+ 9772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:08.351185 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:08.351302 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:08.351395 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:08.353885 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:08.355079 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:08.357623 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:08.358307 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:08.358457 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:08.363769 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:08.363777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:08.363782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:08.363826 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:08.367633 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:08.368801 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:08.368808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:08.368811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:08.369001 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:08.369609 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:08.369614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:08.369618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:08.370009 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:08.370141 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:08.371286 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0619 16:52:08.371594 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21289,"thread_start_us":21478,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
I20260619 16:52:08.373667 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:08.307014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35007 (local address 127.14.153.1:35759)
0619 16:52:08.340174 (+ 33160us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:08.340311 (+ 137us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:08.348965 (+ 8654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:08.352047 (+ 3082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:08.352060 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:08.352750 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:08.353640 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:08.353668 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:08.355287 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:08.355293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:08.357419 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:08.357446 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:08.358796 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:08.358804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:08.359233 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:08.360117 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:08.360207 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:08.363462 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:08.367891 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:08.367902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:08.367923 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:08.368633 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:08.369191 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:08.369196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:08.369199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:08.369466 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:08.370248 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:08.370257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:08.370759 (+ 502us) client_negotiation.cc:769] Sending connection context
0619 16:52:08.371175 (+ 416us) client_negotiation.cc:240] Negotiation successful
0619 16:52:08.371775 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32724,"thread_start_us":42255,"threads_started":1}
I20260619 16:52:08.376003 27505 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35007
I20260619 16:52:08.376374 27505 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:08.377281 27505 heartbeater.cc:511] Master 127.14.153.62:35007 requested a full tablet report, sending...
I20260619 16:52:08.380254 27292 ts_manager.cc:194] Registered new tserver with Master: 945f1a3d1c2044718cda0c80fe639357 (127.14.153.1:45191)
I20260619 16:52:08.382380 27292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:35759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:52:08.407194 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:47457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35007
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:52:08 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887927, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
Jun 19 16:52:08 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887927, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
W20260619 16:52:08.945604 27511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:08.946174 27511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:08.946388 27511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:08.959092 27511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:08.959339 27511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:08.959591 27511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:52:08.976540 27511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.153.20:47457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:08.978583 27511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:08.980479 27511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:09.023732 27518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:09.025784 27519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:09.026376 27511 server_base.cc:1034] running on GCE node
W20260619 16:52:09.027020 27522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:09.027968 27511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:09.034687 27511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:09.036191 27511 hybrid_clock.cc:648] HybridClock initialized: now 1781887929036084 us; error 107 us; skew 500 ppm
I20260619 16:52:09.036695 27511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:09 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:09.041379 27511 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:52:09.062142 27511 webserver.cc:460] Webserver started at http://127.14.153.2:37191/ using document root <none> and password file <none>
I20260619 16:52:09.063396 27511 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:09.063638 27511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:09.064105 27511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:52:09 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:09.093025 27511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "c387c38c1a5a4e6083cbf7f2a4902def"
format_stamp: "Formatted at 2026-06-19 16:52:09 on dist-test-slave-qb36"
server_key: "2f74d1620fed3ebfd9c2fb9a3d2f10e3"
server_key_iv: "5f501bec3163c96f9208e1716d3789fe"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:09.094103 27511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "c387c38c1a5a4e6083cbf7f2a4902def"
format_stamp: "Formatted at 2026-06-19 16:52:09 on dist-test-slave-qb36"
server_key: "2f74d1620fed3ebfd9c2fb9a3d2f10e3"
server_key_iv: "5f501bec3163c96f9208e1716d3789fe"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:09.109895 27511 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.010s sys 0.003s
I20260619 16:52:09.159946 27535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:09.162115 27511 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.015s sys 0.003s
I20260619 16:52:09.162393 27511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "c387c38c1a5a4e6083cbf7f2a4902def"
format_stamp: "Formatted at 2026-06-19 16:52:09 on dist-test-slave-qb36"
server_key: "2f74d1620fed3ebfd9c2fb9a3d2f10e3"
server_key_iv: "5f501bec3163c96f9208e1716d3789fe"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:09.162755 27511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:09.208292 27511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:09.214164 27511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:09.214582 27511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:09.222187 27511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:09.230623 27511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:09.230821 27511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 16:52:09.231009 27511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:09.231150 27511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:09.386411 27505 heartbeater.cc:503] Master 127.14.153.62:35007 was elected leader, sending a full tablet report...
I20260619 16:52:09.477828 27511 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:46255
I20260619 16:52:09.477880 27647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:46255 every 8 connection(s)
I20260619 16:52:09.479607 27511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:52:09.483208 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 27511
I20260619 16:52:09.483383 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:52:09.483922 14948 external_mini_cluster.cc:1428] Setting key 055efb4825c71495f3e8d1b017053ac9
Jun 19 16:52:09 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:09.509538 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:09.483855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:54205 (local address 127.14.153.62:35007)
0619 16:52:09.491561 (+ 7706us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:09.491568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:09.492634 (+ 1066us) server_negotiation.cc:407] Connection header received
0619 16:52:09.494621 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:09.494628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:09.494717 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:09.494857 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:09.496958 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:09.497686 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:09.499328 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:09.499719 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:09.499896 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:09.503624 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:09.503629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:09.503632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:09.503665 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:09.506321 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:09.507076 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:09.507082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:09.507093 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:09.507209 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:09.507788 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:09.507798 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:09.507800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:09.508081 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:09.508238 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:09.508942 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 16:52:09.509162 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7560,"thread_start_us":7411,"threads_started":1}
I20260619 16:52:09.510581 27650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:09.483186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35007 (local address 127.14.153.2:54205)
0619 16:52:09.492120 (+ 8934us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:09.492182 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:09.494326 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:09.495169 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:09.495178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:09.495880 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:09.496785 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:09.496805 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:09.497846 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:09.497851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:09.499182 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:09.499201 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:09.499996 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:09.500003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:09.500235 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:09.500940 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:09.500966 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:09.503385 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:09.506477 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:09.506486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:09.506504 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:09.506921 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:09.507480 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:09.507483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:09.507485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:09.507667 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:09.508210 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:09.508218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:09.508565 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 16:52:09.508830 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 16:52:09.509357 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8593,"thread_start_us":9348,"threads_started":1}
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:52:09.512337 27648 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35007
I20260619 16:52:09.512660 27648 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:09.513443 27648 heartbeater.cc:511] Master 127.14.153.62:35007 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
I20260619 16:52:09.515179 27292 ts_manager.cc:194] Registered new tserver with Master: c387c38c1a5a4e6083cbf7f2a4902def (127.14.153.2:46255)
I20260619 16:52:09.515996 27292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:54205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:52:09.546645 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:47457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35007
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:09.877760 27654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:09.878042 27654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:09.878132 27654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:09.885593 27654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:09.885715 27654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:09.885828 27654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:52:09.895697 27654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.153.20:47457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:09.896978 27654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:09.898276 27654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:09.911478 27659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:09.911568 27660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:09.911595 27662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:09.912475 27654 server_base.cc:1034] running on GCE node
I20260619 16:52:09.913065 27654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:09.914016 27654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:09.915227 27654 hybrid_clock.cc:648] HybridClock initialized: now 1781887929915169 us; error 53 us; skew 500 ppm
I20260619 16:52:09.915549 27654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:09 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:09.918681 27654 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:52:09.920215 27654 webserver.cc:460] Webserver started at http://127.14.153.3:42033/ using document root <none> and password file <none>
I20260619 16:52:09.920898 27654 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:09.921041 27654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:09.921342 27654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:52:09 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:09.940212 27654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "9eecedff355d4431ac9908aaecaf7f37"
format_stamp: "Formatted at 2026-06-19 16:52:09 on dist-test-slave-qb36"
server_key: "00020f610fcb1d554aabd8b46536066d"
server_key_iv: "06d1f11096aa88ac68336f8c1174a215"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:09.941382 27654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "9eecedff355d4431ac9908aaecaf7f37"
format_stamp: "Formatted at 2026-06-19 16:52:09 on dist-test-slave-qb36"
server_key: "00020f610fcb1d554aabd8b46536066d"
server_key_iv: "06d1f11096aa88ac68336f8c1174a215"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:09.948939 27654 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260619 16:52:09.964498 27669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:09.966032 27654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:52:09.966233 27654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "9eecedff355d4431ac9908aaecaf7f37"
format_stamp: "Formatted at 2026-06-19 16:52:09 on dist-test-slave-qb36"
server_key: "00020f610fcb1d554aabd8b46536066d"
server_key_iv: "06d1f11096aa88ac68336f8c1174a215"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:09.966470 27654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:09.987964 27654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:09.991901 27654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:09.992204 27654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:09.993194 27654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:09.994575 27654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:09.994649 27654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:09.994778 27654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:09.994829 27654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:10.028760 27654 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:44237
I20260619 16:52:10.028795 27781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:44237 every 8 connection(s)
I20260619 16:52:10.030225 27654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:10.040627 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 27654
I20260619 16:52:10.040809 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:52:10.041324 14948 external_mini_cluster.cc:1428] Setting key 2a28254b25e1377f6081f29e4f1c2c47
Jun 19 16:52:10 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:10.050544 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:10.033041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:35991 (local address 127.14.153.62:35007)
0619 16:52:10.033554 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:10.033559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:10.034434 (+ 875us) server_negotiation.cc:407] Connection header received
0619 16:52:10.036275 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:10.036282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:10.036398 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:10.036525 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:10.037923 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.038662 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.040316 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.040905 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.041055 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:10.044683 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:10.044690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:10.044695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:10.044742 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:10.047689 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:10.048371 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:10.048375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:10.048377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:10.048541 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:10.049095 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:10.049098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:10.049100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:10.049348 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:10.049470 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:10.050109 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 16:52:10.050290 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":277,"threads_started":1}
I20260619 16:52:10.051654 27785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:10.033482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35007 (local address 127.14.153.3:35991)
0619 16:52:10.034271 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:10.034312 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:10.035999 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:10.036763 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:10.036771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:10.037190 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:10.037753 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:10.037769 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.038826 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.038831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:10.040158 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:10.040177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.041142 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.041151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:10.041426 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:10.042203 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:10.042229 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:10.044480 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:10.047819 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:10.047825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:10.047840 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:10.048261 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:10.048733 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:10.048737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:10.048739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:10.048966 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:10.049454 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:10.049459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:10.049795 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 16:52:10.050028 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 16:52:10.050440 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":274,"threads_started":1}
I20260619 16:52:10.053226 27782 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35007
I20260619 16:52:10.053570 27782 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:10.054298 27782 heartbeater.cc:511] Master 127.14.153.62:35007 requested a full tablet report, sending...
I20260619 16:52:10.056077 27291 ts_manager.cc:194] Registered new tserver with Master: 9eecedff355d4431ac9908aaecaf7f37 (127.14.153.3:44237)
I20260619 16:52:10.057159 27291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:35991
I20260619 16:52:10.057200 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:52:10.059094 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 27372
Jun 19 16:52:10 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:52:10.076270 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:10.060171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.14.153.1:45191)
0619 16:52:10.060723 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:10.060729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:10.060752 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:52:10.060814 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:10.060821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:10.061004 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:10.061132 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:10.062568 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.063698 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.065269 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.065815 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.065914 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:10.070648 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:10.070660 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:10.070674 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:10.070711 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:10.073541 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:10.074210 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:10.074217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:10.074219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:10.074363 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:10.074715 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:10.074719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:10.074720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:10.075124 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:10.075357 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:10.075687 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 16:52:10.075879 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":294,"threads_started":1}
I20260619 16:52:10.077600 27500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:10.161345 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 27372
I20260619 16:52:10.177686 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 27511
Jun 19 16:52:10 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:52:10.195508 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:10.178672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.14.153.2:46255)
0619 16:52:10.179449 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:10.179460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:10.179491 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:52:10.179571 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:10.179581 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:10.179895 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:10.180086 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:10.181732 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.182807 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.184385 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.184962 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.185110 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:10.189481 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:10.189494 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:10.189510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:10.189546 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:10.192520 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:10.193238 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:10.193246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:10.193249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:10.193409 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:10.193945 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:10.193949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:10.193951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:10.194318 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:10.194557 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:10.194887 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 16:52:10.195200 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":480,"threads_started":1}
I20260619 16:52:10.196889 27643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:10.280846 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 27511
I20260619 16:52:10.296657 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 27654
Jun 19 16:52:10 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:52:10.314286 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:10.297995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.14.153.3:44237)
0619 16:52:10.298654 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:10.298662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:10.298691 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:52:10.298749 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:10.298756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:10.298997 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:10.299190 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:10.300553 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.301640 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.303011 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:10.303656 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:10.303759 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:10.308544 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:10.308558 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:10.308574 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:10.308610 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:10.311831 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:10.312528 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:10.312535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:10.312537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:10.312657 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:10.312978 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:10.312981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:10.312983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:10.313299 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:10.313533 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:10.313800 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 16:52:10.314006 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":352,"threads_started":1}
I20260619 16:52:10.315663 27777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:10.400173 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 27654
I20260619 16:52:10.415900 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 27260
I20260619 16:52:10.417277 27322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:10.523896 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 27260
I20260619 16:52:10.549115 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35007
--webserver_interface=127.14.153.62
--webserver_port=45805
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35007
--ranger_config_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:47457/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.153.20:47457/kms with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:10.894551 27794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:10.894845 27794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:10.895012 27794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:10.902381 27794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:10.902483 27794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:10.902529 27794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:10.902581 27794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:10.902628 27794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:10.912799 27794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.153.20:47457/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35007
--ranger_config_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=45805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:10.914062 27794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:10.915436 27794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:10.927525 27800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:10.927469 27802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:10.928025 27799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:10.928689 27794 server_base.cc:1034] running on GCE node
I20260619 16:52:10.929474 27794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:10.931128 27794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:10.932381 27794 hybrid_clock.cc:648] HybridClock initialized: now 1781887930932341 us; error 53 us; skew 500 ppm
I20260619 16:52:10.932744 27794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:10 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887930, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:10.935923 27794 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:52:10.937495 27794 webserver.cc:460] Webserver started at http://127.14.153.62:45805/ using document root <none> and password file <none>
I20260619 16:52:10.938154 27794 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:10.938305 27794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:52:11 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887930, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:11.027547 27794 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 16:52:11.031428 27810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:11.033100 27794 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:52:11.033314 27794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "6fef7229efb6427899bcfbfdca5f3578"
format_stamp: "Formatted at 2026-06-19 16:52:06 on dist-test-slave-qb36"
server_key: "2e33fadeb62c36be4376df0fb9dd1f7b"
server_key_iv: "93ddf0639dc6b9aedd4c61460727c813"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:11.034437 27794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:11.051136 27794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:11.055537 27794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:11.055850 27794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:11.074565 27794 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35007
I20260619 16:52:11.074589 27861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35007 every 8 connection(s)
I20260619 16:52:11.076128 27794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:11.083341 27862 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:52:11.084079 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 27794
I20260619 16:52:11.084969 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:47457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:45191
--local_ip_for_outbound_sockets=127.14.153.1
--tserver_master_addrs=127.14.153.62:35007
--webserver_port=41771
--webserver_interface=127.14.153.1
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.153.20:47457/kms with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:11.086879 27862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Bootstrap starting.
I20260619 16:52:11.107656 27862 log.cc:824] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:11.116827 27862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:11.117393 27862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Bootstrap complete.
I20260619 16:52:11.127204 27862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } }
I20260619 16:52:11.128126 27862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fef7229efb6427899bcfbfdca5f3578, State: Initialized, Role: FOLLOWER
I20260619 16:52:11.128789 27862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } }
I20260619 16:52:11.129046 27862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:11.129184 27862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:11.129367 27862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:52:11.133502 27862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } }
I20260619 16:52:11.134027 27862 leader_election.cc:302] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fef7229efb6427899bcfbfdca5f3578; no voters:
I20260619 16:52:11.134693 27862 leader_election.cc:288] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:52:11.134850 27866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:52:11.136132 27866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [term 2 LEADER]: Becoming Leader. State: Replica: 6fef7229efb6427899bcfbfdca5f3578, State: Running, Role: LEADER
I20260619 16:52:11.136811 27866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } }
I20260619 16:52:11.137247 27862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:11.144693 27868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fef7229efb6427899bcfbfdca5f3578. Latest consensus state: current_term: 2 leader_uuid: "6fef7229efb6427899bcfbfdca5f3578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } } }
I20260619 16:52:11.145226 27868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:11.145844 27867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6fef7229efb6427899bcfbfdca5f3578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef7229efb6427899bcfbfdca5f3578" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35007 } } }
I20260619 16:52:11.146178 27867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:11.150122 27873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:11.156389 27873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:11.157277 27862 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:52:11.158007 27873 catalog_manager.cc:1259] Loaded cluster ID: f97ec72fc422435ebee235290323d8bd
I20260619 16:52:11.158175 27873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:11.162134 27873 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:11.163635 27873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Loaded TSK: 0
I20260619 16:52:11.165237 27873 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:52:11.532020 27864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:11.532547 27864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:11.532754 27864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:11.542834 27864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:11.543072 27864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:11.543304 27864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:52:11.557178 27864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.153.20:47457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:45191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=41771
--tserver_master_addrs=127.14.153.62:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:11.558629 27864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:11.560238 27864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:11.592574 27900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:11.593583 27901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:11.597776 27864 server_base.cc:1034] running on GCE node
W20260619 16:52:11.598089 27904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:11.599161 27864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:11.606665 27864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:11.608035 27864 hybrid_clock.cc:648] HybridClock initialized: now 1781887931607972 us; error 86 us; skew 500 ppm
I20260619 16:52:11.608533 27864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:11 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:11.615422 27864 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:52:11.620299 27864 webserver.cc:460] Webserver started at http://127.14.153.1:41771/ using document root <none> and password file <none>
I20260619 16:52:11.621363 27864 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:11.621556 27864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:52:11 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:11.652951 27864 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.001s
I20260619 16:52:11.672088 27911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:11.674598 27864 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.007s sys 0.003s
I20260619 16:52:11.674945 27864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "945f1a3d1c2044718cda0c80fe639357"
format_stamp: "Formatted at 2026-06-19 16:52:08 on dist-test-slave-qb36"
server_key: "9541d397bad264402954e5fdf63d6e30"
server_key_iv: "7dde660e93f9486573b5392db88d025f"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:11.676628 27864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:11.710566 27864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:11.716074 27864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:11.716573 27864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:11.732904 27864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:11.740679 27864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:11.740844 27864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:11.741034 27864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:11.741209 27864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:11.937316 27864 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:45191
I20260619 16:52:11.937378 28024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:45191 every 8 connection(s)
I20260619 16:52:11.939510 27864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:52:11.940464 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 27864
I20260619 16:52:11.941277 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:47457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:46255
--local_ip_for_outbound_sockets=127.14.153.2
--tserver_master_addrs=127.14.153.62:35007
--webserver_port=37191
--webserver_interface=127.14.153.2
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.153.20:47457/kms with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:52:11 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:12.006145 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:11.950689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35007 (local address 127.14.153.1:46389)
0619 16:52:11.975658 (+ 24969us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:11.975730 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:11.978157 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:11.984028 (+ 5871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:11.984040 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:11.985069 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:11.985887 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:11.985916 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:11.988447 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:11.988454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:11.990453 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:11.990471 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:11.992308 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:11.992315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:11.992594 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:11.993416 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:11.993446 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:11.996254 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:12.000727 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:12.000737 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:12.000756 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:12.001356 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:12.002001 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:12.002006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:12.002010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:12.002279 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:12.003377 (+ 1098us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:12.003384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:12.003821 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 16:52:12.004113 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 16:52:12.004702 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24561,"thread_start_us":27776,"threads_started":1}
I20260619 16:52:12.008642 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:11.952188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:46389 (local address 127.14.153.62:35007)
0619 16:52:11.980199 (+ 28011us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:11.980220 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:11.980271 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:52:11.980725 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:11.980746 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:11.981277 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:11.981730 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:11.986181 (+ 4451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:11.988201 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:11.990658 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:11.991714 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:11.991904 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:11.996655 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:11.996666 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:11.996688 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:11.996740 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:12.000526 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:12.001554 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:12.001564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:12.001573 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:12.001860 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:12.002411 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:12.002416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:12.002419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:12.003113 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:12.003294 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:12.006886 (+ 3592us) server_negotiation.cc:299] Negotiation successful
0619 16:52:12.007342 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27686,"thread_start_us":29799,"threads_started":1}
I20260619 16:52:12.013048 28025 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35007
I20260619 16:52:12.013429 28025 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:12.014295 28025 heartbeater.cc:511] Master 127.14.153.62:35007 requested a full tablet report, sending...
W20260619 16:52:12.017735 28025 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:35007 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:52:12.017961 28025 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:12.018239 28025 heartbeater.cc:511] Master 127.14.153.62:35007 requested a full tablet report, sending...
I20260619 16:52:12.019676 28025 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:12.020026 28025 heartbeater.cc:511] Master 127.14.153.62:35007 requested a full tablet report, sending...
W20260619 16:52:12.021126 28025 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:52:12.250314 28036 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 16:52:12.427073 28027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:12.427603 28027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:12.427834 28027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:12.439693 28027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:12.439929 28027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:12.440158 28027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:52:12.455933 28027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.153.20:47457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:46255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=37191
--tserver_master_addrs=127.14.153.62:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:12.457624 28027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:12.459473 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:12.491817 28045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:12.492576 28046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:12.500514 28048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:12.504716 28027 server_base.cc:1034] running on GCE node
I20260619 16:52:12.505546 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:12.506829 28027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:12.508118 28027 hybrid_clock.cc:648] HybridClock initialized: now 1781887932508042 us; error 65 us; skew 500 ppm
I20260619 16:52:12.508518 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:12 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:12.512856 28027 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:52:12.514938 28027 webserver.cc:460] Webserver started at http://127.14.153.2:37191/ using document root <none> and password file <none>
I20260619 16:52:12.515985 28027 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:12.516155 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:52:12 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
Jun 19 16:52:12 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:12.544268 28027 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:52:12.547525 28056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:12.549542 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260619 16:52:12.549800 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "c387c38c1a5a4e6083cbf7f2a4902def"
format_stamp: "Formatted at 2026-06-19 16:52:09 on dist-test-slave-qb36"
server_key: "2f74d1620fed3ebfd9c2fb9a3d2f10e3"
server_key_iv: "5f501bec3163c96f9208e1716d3789fe"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:12.551507 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:12.597432 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:12.601930 28027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:12.602227 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:12.603504 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:12.605180 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:12.605310 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:12.605453 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:12.605543 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:12.653340 28027 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:46255
I20260619 16:52:12.653371 28169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:46255 every 8 connection(s)
I20260619 16:52:12.655324 28027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:52:12.663398 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 28027
I20260619 16:52:12.664208 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.153.20:47457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:44237
--local_ip_for_outbound_sockets=127.14.153.3
--tserver_master_addrs=127.14.153.62:35007
--webserver_port=42033
--webserver_interface=127.14.153.3
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.153.20:47457/kms with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:52:12 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:12.684007 28172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:12.658162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:57795 (local address 127.14.153.62:35007)
0619 16:52:12.658740 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:12.658748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:12.659861 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 16:52:12.662608 (+ 2747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:12.662616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:12.662718 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:12.662846 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:12.665714 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:12.666960 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:12.669677 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:12.670366 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:12.670471 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:12.675299 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:12.675308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:12.675317 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:12.675367 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:12.679421 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:12.680545 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:12.680552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:12.680556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:12.680718 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:12.681301 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:12.681309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:12.681315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:12.681755 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:12.681914 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:12.683117 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0619 16:52:12.683429 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":292,"threads_started":1}
I20260619 16:52:12.685164 28173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:12.658791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35007 (local address 127.14.153.2:57795)
0619 16:52:12.659680 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:12.659728 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:12.662227 (+ 2499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:12.663181 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:12.663838 (+ 657us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:12.664509 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:12.665447 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:12.665479 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:12.667243 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:12.667251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:12.669402 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:12.669419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:12.670888 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:12.670898 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:12.671270 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:12.672155 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:12.672196 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:12.674975 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:12.679604 (+ 4629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:12.679614 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:12.679637 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:12.680348 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:12.680860 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:12.680865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:12.680869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:12.681145 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:12.681914 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:12.681923 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:12.682513 (+ 590us) client_negotiation.cc:769] Sending connection context
0619 16:52:12.682907 (+ 394us) client_negotiation.cc:240] Negotiation successful
0619 16:52:12.683623 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":366,"threads_started":1}
I20260619 16:52:12.687019 28170 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35007
I20260619 16:52:12.687462 28170 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:12.688493 28170 heartbeater.cc:511] Master 127.14.153.62:35007 requested a full tablet report, sending...
I20260619 16:52:12.690979 27827 ts_manager.cc:194] Registered new tserver with Master: c387c38c1a5a4e6083cbf7f2a4902def (127.14.153.2:46255)
I20260619 16:52:12.693586 27827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:57795
Jun 19 16:52:12 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:13.021286 28025 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:13.021739 28025 heartbeater.cc:511] Master 127.14.153.62:35007 requested a full tablet report, sending...
I20260619 16:52:13.022987 27827 ts_manager.cc:194] Registered new tserver with Master: 945f1a3d1c2044718cda0c80fe639357 (127.14.153.1:45191)
I20260619 16:52:13.024171 27827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:46389
W20260619 16:52:13.137391 28174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:13.137846 28174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:13.138065 28174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:13.149523 28174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:13.149685 28174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:13.149926 28174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:52:13.165879 28174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.153.20:47457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:44237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=42033
--tserver_master_addrs=127.14.153.62:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:13.167712 28174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:13.169480 28174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:13.187721 28180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:13.187736 28179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:13.195317 28174 server_base.cc:1034] running on GCE node
W20260619 16:52:13.196406 28182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:13.197329 28174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:13.198697 28174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:13.200021 28174 hybrid_clock.cc:648] HybridClock initialized: now 1781887933199943 us; error 66 us; skew 500 ppm
I20260619 16:52:13.200433 28174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:13 dist-test-slave-qb36 krb5kdc[26816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:13.204954 28174 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:52:13.208606 28174 webserver.cc:460] Webserver started at http://127.14.153.3:42033/ using document root <none> and password file <none>
I20260619 16:52:13.209637 28174 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:13.209813 28174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:52:13 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:13.245564 28174 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.004s sys 0.004s
I20260619 16:52:13.249593 28190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:13.251725 28174 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:52:13.251952 28174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "9eecedff355d4431ac9908aaecaf7f37"
format_stamp: "Formatted at 2026-06-19 16:52:09 on dist-test-slave-qb36"
server_key: "00020f610fcb1d554aabd8b46536066d"
server_key_iv: "06d1f11096aa88ac68336f8c1174a215"
server_key_version: "kuduclusterkey@0"
I20260619 16:52:13.253455 28174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:13.277550 28174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:13.283092 28174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:13.283545 28174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:13.292213 28174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:13.299292 28174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:13.299428 28174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:13.299587 28174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:13.299674 28174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:52:13 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887932, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for HTTP/127.14.153.20@KRBTEST.COM
I20260619 16:52:13.503465 28174 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:44237
I20260619 16:52:13.503551 28303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:44237 every 8 connection(s)
I20260619 16:52:13.505410 28174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:13.510265 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 28174
Jun 19 16:52:13 dist-test-slave-qb36 krb5kdc[26816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:13.548481 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.511711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.14.153.62:35007)
0619 16:52:13.524170 (+ 12459us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.524176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.524199 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:52:13.524270 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.524276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.524380 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.524499 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:13.527821 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.528996 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.531087 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.531745 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.531877 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.535484 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:13.535494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:13.535498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:13.535536 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:13.539746 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:13.540722 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:13.540727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:13.540731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:13.540878 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:13.541366 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:13.541372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:13.541375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:13.541781 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:13.542046 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.547791 (+ 5745us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.548064 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12329,"thread_start_us":10208,"threads_started":1}
I20260619 16:52:13.552819 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.509614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:34349 (local address 127.14.153.62:35007)
0619 16:52:13.522202 (+ 12588us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.522210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.527720 (+ 5510us) server_negotiation.cc:407] Connection header received
0619 16:52:13.527793 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.527798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.527907 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.528102 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:13.530697 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.532372 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.535058 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.535786 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.535919 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.543399 (+ 7480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:13.543406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:13.543411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:13.543446 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:13.546838 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:13.547885 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:13.547890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:13.547894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:13.548036 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:13.548752 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:13.548757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:13.548760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:13.549116 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:13.549323 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.550544 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.551060 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12436,"thread_start_us":14280,"threads_started":1}
I20260619 16:52:13.553862 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.509040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35007 (local address 127.14.153.3:34349)
0619 16:52:13.524642 (+ 15602us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:13.524705 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:13.527293 (+ 2588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:13.528808 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:13.528823 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:13.529558 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:13.530464 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.530494 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.532556 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.532562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.534832 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.534849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.536295 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.536303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.536597 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.538284 (+ 1687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:13.538316 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:13.542010 (+ 3694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:13.547065 (+ 5055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:13.547076 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:13.547097 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:13.547731 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:13.548295 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:13.548300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:13.548316 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:13.548605 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:13.549397 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:13.549406 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:13.550049 (+ 643us) client_negotiation.cc:769] Sending connection context
0619 16:52:13.551508 (+ 1459us) client_negotiation.cc:240] Negotiation successful
0619 16:52:13.552107 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15242,"thread_start_us":15499,"threads_started":1}
I20260619 16:52:13.555522 28304 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35007
I20260619 16:52:13.555887 28304 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:13.556828 28304 heartbeater.cc:511] Master 127.14.153.62:35007 requested a full tablet report, sending...
I20260619 16:52:13.558691 27827 ts_manager.cc:194] Registered new tserver with Master: 9eecedff355d4431ac9908aaecaf7f37 (127.14.153.3:44237)
I20260619 16:52:13.559801 27827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:34349
I20260619 16:52:13.565898 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:52:13.583534 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.569263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.14.153.62:35007)
0619 16:52:13.569432 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.569437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.569930 (+ 493us) server_negotiation.cc:407] Connection header received
0619 16:52:13.570142 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.570147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.570224 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.570332 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:13.572103 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.572827 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.574178 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.574594 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.574746 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.577424 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:13.577429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:13.577433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:13.577468 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:13.580677 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:13.581379 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:13.581385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:13.581389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:13.581553 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:13.581934 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:13.581940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:13.581943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:13.582274 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:13.582541 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.582901 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.583174 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:52:13.589385 27827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:52:13.600584 27827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:52:13.634131 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.616798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.0.0.1:42844)
0619 16:52:13.617571 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:13.617615 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:13.617790 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:13.621060 (+ 3270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:13.621070 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:13.621111 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:13.621620 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.621636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.625668 (+ 4032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.625677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.630182 (+ 4505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.630203 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.632774 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.632783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.633089 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.633172 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 16:52:13.633309 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 16:52:13.633565 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":386,"threads_started":1}
I20260619 16:52:13.637365 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.617123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.14.153.1:45191)
0619 16:52:13.620293 (+ 3170us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.620306 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.620342 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:52:13.620465 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.620477 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.620777 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.621851 (+ 1074us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:13.622136 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.625471 (+ 3335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.630454 (+ 4983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.632483 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.636182 (+ 3699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.636566 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.636663 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.636900 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1738,"thread_start_us":5139,"threads_started":1}
I20260619 16:52:13.643697 27960 tablet_service.cc:1467] Processing CreateTablet for tablet 57401753b2bf466793307bd6518c6e68 (DEFAULT_TABLE table=test-table [id=d9a763d78bda4259873c21117a4ffbc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:13.645412 27960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57401753b2bf466793307bd6518c6e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:13.647354 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.618319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:46255 (local address 127.0.0.1:47898)
0619 16:52:13.634404 (+ 16085us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:13.634423 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:13.634532 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:13.640456 (+ 5924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:13.640462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:13.640494 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:13.640699 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.640709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.642730 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.642735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.644563 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.644587 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.646601 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.646607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.646788 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.646828 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:52:13.646896 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:52:13.647069 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15992,"thread_start_us":20918,"threads_started":1}
I20260619 16:52:13.648526 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.618840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.14.153.2:46255)
0619 16:52:13.639780 (+ 20940us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.639789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.639818 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:52:13.639938 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.639945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.640181 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.640870 (+ 689us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:13.641087 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.642578 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.645008 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.646426 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.647787 (+ 1361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.647991 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.648057 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.648217 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20772,"threads_started":1}
I20260619 16:52:13.650857 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.618796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:44237 (local address 127.0.0.1:41158)
0619 16:52:13.638889 (+ 20093us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:13.638911 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:13.639222 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:13.640321 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:13.640326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:13.640351 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:13.642980 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.642992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.644989 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.644994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.647085 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.647102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.649949 (+ 2847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.649955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.650134 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.650166 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:52:13.650253 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:52:13.650413 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19936,"mutex_wait_us":2199,"spinlock_wait_cycles":3328,"thread_start_us":19778,"threads_started":1}
I20260619 16:52:13.650866 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.619434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.14.153.3:44237)
0619 16:52:13.638386 (+ 18952us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.638399 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.639089 (+ 690us) server_negotiation.cc:407] Connection header received
0619 16:52:13.639484 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.639493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.639802 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.640021 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:13.643232 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.644834 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.647273 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.649712 (+ 2439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.649860 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.650099 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.650349 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.650531 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18676,"thread_start_us":18438,"threads_started":1}
I20260619 16:52:13.655387 28239 tablet_service.cc:1467] Processing CreateTablet for tablet 57401753b2bf466793307bd6518c6e68 (DEFAULT_TABLE table=test-table [id=d9a763d78bda4259873c21117a4ffbc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:13.657145 28239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57401753b2bf466793307bd6518c6e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:13.661412 28105 tablet_service.cc:1467] Processing CreateTablet for tablet 57401753b2bf466793307bd6518c6e68 (DEFAULT_TABLE table=test-table [id=d9a763d78bda4259873c21117a4ffbc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:13.662952 28105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57401753b2bf466793307bd6518c6e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:13.663843 28332 tablet_bootstrap.cc:492] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357: Bootstrap starting.
I20260619 16:52:13.667522 28332 tablet_bootstrap.cc:654] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:13.668846 28332 log.cc:824] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:13.678382 28332 tablet_bootstrap.cc:492] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357: No bootstrap required, opened a new log
I20260619 16:52:13.678823 28332 ts_tablet_manager.cc:1397] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260619 16:52:13.689338 28332 raft_consensus.cc:348] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } }
I20260619 16:52:13.690012 28332 raft_consensus.cc:374] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:13.690163 28332 raft_consensus.cc:729] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945f1a3d1c2044718cda0c80fe639357, State: Initialized, Role: FOLLOWER
I20260619 16:52:13.690552 28335 tablet_bootstrap.cc:492] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: Bootstrap starting.
I20260619 16:52:13.690984 28332 consensus_queue.cc:260] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } }
I20260619 16:52:13.691824 28334 tablet_bootstrap.cc:492] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: Bootstrap starting.
I20260619 16:52:13.695310 28335 tablet_bootstrap.cc:654] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:13.695852 28025 heartbeater.cc:503] Master 127.14.153.62:35007 was elected leader, sending a full tablet report...
I20260619 16:52:13.696337 28334 tablet_bootstrap.cc:654] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:13.696604 28332 ts_tablet_manager.cc:1428] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357: Time spent starting tablet: real 0.018s user 0.013s sys 0.002s
I20260619 16:52:13.696759 28335 log.cc:824] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:13.697536 28334 log.cc:824] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:13.698626 28170 heartbeater.cc:503] Master 127.14.153.62:35007 was elected leader, sending a full tablet report...
I20260619 16:52:13.706259 28335 tablet_bootstrap.cc:492] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: No bootstrap required, opened a new log
I20260619 16:52:13.706586 28335 ts_tablet_manager.cc:1397] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.010s
I20260619 16:52:13.708868 28334 tablet_bootstrap.cc:492] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: No bootstrap required, opened a new log
I20260619 16:52:13.709188 28334 ts_tablet_manager.cc:1397] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260619 16:52:13.717638 28335 raft_consensus.cc:348] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } }
I20260619 16:52:13.718149 28335 raft_consensus.cc:374] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:13.718255 28335 raft_consensus.cc:729] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c387c38c1a5a4e6083cbf7f2a4902def, State: Initialized, Role: FOLLOWER
I20260619 16:52:13.718971 28335 consensus_queue.cc:260] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } }
I20260619 16:52:13.719741 28334 raft_consensus.cc:348] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } }
I20260619 16:52:13.720240 28334 raft_consensus.cc:374] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:13.720387 28334 raft_consensus.cc:729] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eecedff355d4431ac9908aaecaf7f37, State: Initialized, Role: FOLLOWER
I20260619 16:52:13.721186 28334 consensus_queue.cc:260] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } }
I20260619 16:52:13.722788 28335 ts_tablet_manager.cc:1428] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260619 16:52:13.723047 28304 heartbeater.cc:503] Master 127.14.153.62:35007 was elected leader, sending a full tablet report...
I20260619 16:52:13.724581 28334 ts_tablet_manager.cc:1428] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260619 16:52:13.734766 28026 tablet.cc:2369] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:52:13.763164 28305 tablet.cc:2369] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:13.763762 28340 raft_consensus.cc:482] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:13.764083 28340 raft_consensus.cc:504] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } }
I20260619 16:52:13.765961 28340 leader_election.cc:288] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 945f1a3d1c2044718cda0c80fe639357 (127.14.153.1:45191), c387c38c1a5a4e6083cbf7f2a4902def (127.14.153.2:46255)
I20260619 16:52:13.772853 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.766300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.14.153.3:49423)
0619 16:52:13.766538 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:13.766558 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:13.766691 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:13.767342 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:13.767350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:13.767397 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:13.767663 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.767675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.768913 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.768918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.770917 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.770932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.772144 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.772150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.772318 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.772342 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:52:13.772448 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:52:13.772580 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":26}
I20260619 16:52:13.772843 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.766285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:49423 (local address 127.14.153.1:45191)
0619 16:52:13.766551 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.766559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.766638 (+ 79us) server_negotiation.cc:407] Connection header received
0619 16:52:13.766857 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.766865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.766953 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.767094 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:13.767835 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.768770 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.771108 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.771969 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.772144 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.772218 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.772475 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.772599 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:52:13.773692 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.767194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:35223 (local address 127.14.153.2:46255)
0619 16:52:13.767392 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.767397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.767947 (+ 550us) server_negotiation.cc:407] Connection header received
0619 16:52:13.768057 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.768062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.768151 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.768245 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:13.768808 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.769563 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.771336 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.772487 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.772797 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.772868 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.773318 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.773463 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:52:13.774143 27980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57401753b2bf466793307bd6518c6e68" candidate_uuid: "9eecedff355d4431ac9908aaecaf7f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945f1a3d1c2044718cda0c80fe639357" is_pre_election: true
I20260619 16:52:13.774389 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.766785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:46255 (local address 127.14.153.3:35223)
0619 16:52:13.767817 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:13.767833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:13.767954 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:13.768273 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:13.768278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:13.768301 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:13.768640 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.768650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.769708 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.769714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.771187 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:13.771201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.772989 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.772995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:13.773150 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.773179 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:52:13.773912 (+ 733us) client_negotiation.cc:240] Negotiation successful
0619 16:52:13.774042 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":43,"thread_start_us":742,"threads_started":1}
I20260619 16:52:13.774714 27980 raft_consensus.cc:2454] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eecedff355d4431ac9908aaecaf7f37 in term 0.
I20260619 16:52:13.775617 28193 leader_election.cc:302] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 945f1a3d1c2044718cda0c80fe639357, 9eecedff355d4431ac9908aaecaf7f37; no voters:
I20260619 16:52:13.776048 28340 raft_consensus.cc:2781] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:52:13.776224 28340 raft_consensus.cc:482] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:52:13.776352 28340 raft_consensus.cc:3037] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:13.776842 28125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57401753b2bf466793307bd6518c6e68" candidate_uuid: "9eecedff355d4431ac9908aaecaf7f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" is_pre_election: true
I20260619 16:52:13.777354 28125 raft_consensus.cc:2454] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eecedff355d4431ac9908aaecaf7f37 in term 0.
I20260619 16:52:13.778846 28340 raft_consensus.cc:504] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } }
I20260619 16:52:13.779556 28340 leader_election.cc:288] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [CANDIDATE]: Term 1 election: Requested vote from peers 945f1a3d1c2044718cda0c80fe639357 (127.14.153.1:45191), c387c38c1a5a4e6083cbf7f2a4902def (127.14.153.2:46255)
I20260619 16:52:13.780014 27980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57401753b2bf466793307bd6518c6e68" candidate_uuid: "9eecedff355d4431ac9908aaecaf7f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "945f1a3d1c2044718cda0c80fe639357"
I20260619 16:52:13.780230 27980 raft_consensus.cc:3037] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:13.780360 28125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57401753b2bf466793307bd6518c6e68" candidate_uuid: "9eecedff355d4431ac9908aaecaf7f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c387c38c1a5a4e6083cbf7f2a4902def"
I20260619 16:52:13.780566 28125 raft_consensus.cc:3037] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:13.783931 27980 raft_consensus.cc:2454] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eecedff355d4431ac9908aaecaf7f37 in term 1.
I20260619 16:52:13.783974 28125 raft_consensus.cc:2454] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eecedff355d4431ac9908aaecaf7f37 in term 1.
I20260619 16:52:13.784530 28193 leader_election.cc:302] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9eecedff355d4431ac9908aaecaf7f37, c387c38c1a5a4e6083cbf7f2a4902def; no voters:
I20260619 16:52:13.784854 28340 raft_consensus.cc:2781] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:13.785786 28340 raft_consensus.cc:686] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 1 LEADER]: Becoming Leader. State: Replica: 9eecedff355d4431ac9908aaecaf7f37, State: Running, Role: LEADER
I20260619 16:52:13.786739 28340 consensus_queue.cc:237] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } }
I20260619 16:52:13.792245 27826 catalog_manager.cc:5495] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9eecedff355d4431ac9908aaecaf7f37 (127.14.153.3). New cstate: current_term: 1 leader_uuid: "9eecedff355d4431ac9908aaecaf7f37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eecedff355d4431ac9908aaecaf7f37" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 44237 } health_report { overall_health: HEALTHY } } }
I20260619 16:52:13.826646 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.820691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.14.153.3:44237)
0619 16:52:13.820912 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.820918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.820939 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:52:13.821048 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.821053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.821157 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.821309 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:52:13.821678 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.822738 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.824284 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.824857 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.824975 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.825433 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:52:13.826092 (+ 659us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:52:13.826229 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.826272 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.826383 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:52:13.835556 28125 raft_consensus.cc:1261] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [term 1 FOLLOWER]: Refusing update from remote peer 9eecedff355d4431ac9908aaecaf7f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:13.835613 27980 raft_consensus.cc:1261] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [term 1 FOLLOWER]: Refusing update from remote peer 9eecedff355d4431ac9908aaecaf7f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:13.836766 28342 consensus_queue.cc:1035] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c387c38c1a5a4e6083cbf7f2a4902def" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 46255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:13.837180 28340 consensus_queue.cc:1035] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "945f1a3d1c2044718cda0c80fe639357" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:13.853998 28345 mvcc.cc:205] Tried to move back new op lower bound from 7298612976978649088 to 7298612976795561984. Current Snapshot: MvccSnapshot[applied={T|T < 7298612976978649088}]
I20260619 16:52:13.855130 28347 mvcc.cc:205] Tried to move back new op lower bound from 7298612976978649088 to 7298612976795561984. Current Snapshot: MvccSnapshot[applied={T|T < 7298612976978649088}]
I20260619 16:52:13.886710 27826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:13.887063 27826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:13.892323 27826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fef7229efb6427899bcfbfdca5f3578: Sending DeleteTablet for 3 replicas of tablet 57401753b2bf466793307bd6518c6e68
I20260619 16:52:13.893848 27960 tablet_service.cc:1514] Processing DeleteTablet for tablet 57401753b2bf466793307bd6518c6e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:13 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:42844
I20260619 16:52:13.894191 28239 tablet_service.cc:1514] Processing DeleteTablet for tablet 57401753b2bf466793307bd6518c6e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:13 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:41158
I20260619 16:52:13.894965 28105 tablet_service.cc:1514] Processing DeleteTablet for tablet 57401753b2bf466793307bd6518c6e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:13 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:47898
I20260619 16:52:13.907847 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 27864
I20260619 16:52:13.929801 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.909649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.14.153.1:45191)
0619 16:52:13.909835 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:13.909841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:13.909861 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:52:13.909928 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:13.909933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:13.910022 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:13.910151 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:13.913110 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.914457 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.916683 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:13.917538 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:13.917693 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:13.921031 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:13.921049 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:13.921075 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:13.921121 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:13.925887 (+ 4766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:13.926918 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:13.926931 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:13.926937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:13.927146 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:13.927564 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:13.927571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:13.927576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:13.928142 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:13.928510 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:13.928920 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 16:52:13.929217 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:52:13.930589 28020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:13.931365 28357 tablet_replica.cc:330] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: stopping tablet replica
I20260619 16:52:13.932166 28357 raft_consensus.cc:2229] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:13.932919 28356 tablet_replica.cc:330] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: stopping tablet replica
I20260619 16:52:13.933637 28357 raft_consensus.cc:2258] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:13.934305 28356 raft_consensus.cc:2229] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:52:13.937429 28356 raft_consensus.cc:2258] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:13.939824 28357 ts_tablet_manager.cc:1905] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:13.950412 28356 ts_tablet_manager.cc:1905] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:13.961385 28356 ts_tablet_manager.cc:1918] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:13.961735 28357 ts_tablet_manager.cc:1918] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:13.961982 28357 log.cc:1196] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/57401753b2bf466793307bd6518c6e68
I20260619 16:52:13.962606 28357 ts_tablet_manager.cc:1939] T 57401753b2bf466793307bd6518c6e68 P c387c38c1a5a4e6083cbf7f2a4902def: Deleting consensus metadata
I20260619 16:52:13.964900 27813 catalog_manager.cc:4841] TS c387c38c1a5a4e6083cbf7f2a4902def (127.14.153.2:46255): tablet 57401753b2bf466793307bd6518c6e68 (table test-table [id=d9a763d78bda4259873c21117a4ffbc3]) successfully deleted
I20260619 16:52:13.965405 28356 log.cc:1196] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/57401753b2bf466793307bd6518c6e68
I20260619 16:52:13.966331 28356 ts_tablet_manager.cc:1939] T 57401753b2bf466793307bd6518c6e68 P 9eecedff355d4431ac9908aaecaf7f37: Deleting consensus metadata
I20260619 16:52:13.968310 27811 catalog_manager.cc:4841] TS 9eecedff355d4431ac9908aaecaf7f37 (127.14.153.3:44237): tablet 57401753b2bf466793307bd6518c6e68 (table test-table [id=d9a763d78bda4259873c21117a4ffbc3]) successfully deleted
I20260619 16:52:14.051389 28355 tablet_replica.cc:330] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357: stopping tablet replica
I20260619 16:52:14.051900 28355 raft_consensus.cc:2229] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:14.052444 28355 raft_consensus.cc:2258] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:14.054982 28355 ts_tablet_manager.cc:1905] T 57401753b2bf466793307bd6518c6e68 P 945f1a3d1c2044718cda0c80fe639357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:14.056434 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 27864
W20260619 16:52:14.080498 27813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.153.1:45191 (error 108)
I20260619 16:52:14.080724 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 28027
I20260619 16:52:14.082760 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.082230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.0.0.1:42866)
0619 16:52:14.082470 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.082582 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260619 16:52:14.083125 27813 catalog_manager.cc:4584] TS 945f1a3d1c2044718cda0c80fe639357 (127.14.153.1:45191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57401753b2bf466793307bd6518c6e68: Network error: Client connection negotiation failed: client connection to 127.14.153.1:45191: connect: Connection refused (error 111)
I20260619 16:52:14.098465 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.081506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.14.153.2:46255)
0619 16:52:14.081695 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:14.081701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:14.081800 (+ 99us) server_negotiation.cc:407] Connection header received
0619 16:52:14.081993 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:14.082005 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:14.082070 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:14.082180 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:14.084027 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:14.085163 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:14.087229 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:14.087866 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:14.088013 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:14.091352 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:14.091370 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:14.091390 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:14.091438 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:14.095436 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:14.096150 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:14.096160 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:14.096166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:14.096348 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:14.096758 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:14.096763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:14.096766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:14.097262 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:14.097555 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:14.097832 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 16:52:14.098069 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:52:14.099457 28165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:14.136906 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.136333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.0.0.1:42870)
0619 16:52:14.136573 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.136701 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 16:52:14.138258 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.137793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.0.0.1:42884)
0619 16:52:14.137982 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.138099 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 16:52:14.191217 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 28027
I20260619 16:52:14.198750 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.198272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.0.0.1:42888)
0619 16:52:14.198488 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.198584 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 16:52:14.199883 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.199531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.0.0.1:42898)
0619 16:52:14.199682 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.199766 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:52:14.216120 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 28174
I20260619 16:52:14.230854 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.217121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.14.153.3:44237)
0619 16:52:14.217319 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:14.217326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:14.217347 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:52:14.217449 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:14.217455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:14.217549 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:14.217670 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:14.218987 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:14.219873 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:14.221239 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:14.221691 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:14.221843 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:14.224655 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:14.224671 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:14.224687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:14.224726 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:14.227965 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:14.228663 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:14.228680 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:14.228683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:14.228818 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:14.229237 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:14.229241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:14.229246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:14.229604 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:14.229914 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:14.230256 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:52:14.230477 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:52:14.231648 28299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:14.280299 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.279761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.0.0.1:42908)
0619 16:52:14.279993 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.280082 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 16:52:14.281445 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.281046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.0.0.1:42920)
0619 16:52:14.281262 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.281334 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 16:52:14.355618 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 28174
I20260619 16:52:14.371908 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 27794
I20260619 16:52:14.373189 27857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:14.486162 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.485434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:45191 (local address 127.0.0.1:42924)
0619 16:52:14.485721 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.485882 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:45191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260619 16:52:14.486419 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 27794
I20260619 16:52:14.510773 14948 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:52:14.846518 14948 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:52:14.846705 14948 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:52:15.192354 14948 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:52:15.192 UTC [26846] LOG: received smart shutdown request
2026-06-19 16:52:15.195 UTC [26846] LOG: background worker "logical replication launcher" (PID 26855) exited with exit code 1
2026-06-19 16:52:15.196 UTC [26850] LOG: shutting down
2026-06-19 16:52:15.374 UTC [26846] LOG: database system is shut down
2026-06-19T16:52:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53160 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:52:15 dist-test-slave-qb36 krb5kdc[28371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:52:15 dist-test-slave-qb36 krb5kdc[28371](info): set up 2 sockets
Jun 19 16:52:15 dist-test-slave-qb36 krb5kdc[28371](info): commencing operation
krb5kdc: starting...
W20260619 16:52:16.611572 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:52:16 dist-test-slave-qb36 krb5kdc[28371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:52:16.846949 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34173
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34173,127.14.153.61:45755,127.14.153.60:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:17.180927 28384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:17.181305 28384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:17.181497 28384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:17.188947 28384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:17.189065 28384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:17.189182 28384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:17.189291 28384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:17.189373 28384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:17.199478 28384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34173,127.14.153.61:45755,127.14.153.60:35649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:17.200915 28384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:17.202296 28384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:17.216495 28392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.216562 28389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.216590 28390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:17.217613 28384 server_base.cc:1034] running on GCE node
I20260619 16:52:17.218605 28384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:17.220178 28384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:17.221426 28384 hybrid_clock.cc:648] HybridClock initialized: now 1781887937221377 us; error 46 us; skew 500 ppm
I20260619 16:52:17.221792 28384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:17 dist-test-slave-qb36 krb5kdc[28371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:17.228688 28384 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:52:17.230202 28384 webserver.cc:460] Webserver started at http://127.14.153.62:33771/ using document root <none> and password file <none>
I20260619 16:52:17.230924 28384 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:17.231083 28384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:17.231379 28384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:17.234408 28384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "884ba9b377cd423ebd22d0b028c98d87"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-qb36"
server_key: "17e5dcd3a17e10027d7c373b132acd23"
server_key_iv: "170e554ddaeb8f36f96e145f29fb8980"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.235275 28384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "884ba9b377cd423ebd22d0b028c98d87"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-qb36"
server_key: "17e5dcd3a17e10027d7c373b132acd23"
server_key_iv: "170e554ddaeb8f36f96e145f29fb8980"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.240573 28384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 16:52:17.244184 28399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.245764 28384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:52:17.245975 28384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "884ba9b377cd423ebd22d0b028c98d87"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-qb36"
server_key: "17e5dcd3a17e10027d7c373b132acd23"
server_key_iv: "170e554ddaeb8f36f96e145f29fb8980"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.246197 28384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:17.259318 28384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:17.263723 28384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:17.264030 28384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:17.282645 28384 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:34173
I20260619 16:52:17.282665 28450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:34173 every 8 connection(s)
I20260619 16:52:17.284174 28384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:17.288043 28451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:17.289676 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 28384
I20260619 16:52:17.289937 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:52:17.290602 14948 external_mini_cluster.cc:1428] Setting key 3dcff6f98b543a2857561d113900e709
I20260619 16:52:17.294857 28451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:52:17 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.61@KRBTEST.COM" created.
I20260619 16:52:17.318369 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.298131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34173 (local address 127.0.0.1:40896)
0619 16:52:17.299047 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.299102 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.301099 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.301926 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.301933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.302640 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:17.302977 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.302995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.304587 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.304592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.305689 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.305701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.306649 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.306656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.306817 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.307584 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:17.307622 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:17.310689 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:17.314181 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.314192 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.314202 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.314612 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.315137 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.315141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.315143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.315337 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.316428 (+ 1091us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:17.316435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:17.316630 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.316846 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.317267 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":412,"threads_started":1}
I20260619 16:52:17.318411 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.298131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.14.153.62:34173)
0619 16:52:17.299855 (+ 1724us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.299868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.299922 (+ 54us) server_negotiation.cc:407] Connection header received
0619 16:52:17.301469 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.301475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.301697 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.301898 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:17.303225 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.304422 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.305847 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.306410 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.306610 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.310949 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:17.310981 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:17.310999 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:17.311086 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:17.313991 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.314774 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.314785 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.314792 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.314988 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.315477 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.315485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.315491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.316262 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:17.316483 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.317056 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.317494 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":1515,"thread_start_us":1305,"threads_started":1}
I20260619 16:52:17.321729 28451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:52:17.323146 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.322612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46874)
0619 16:52:17.322815 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.322959 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 16:52:17.323473 28403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
I20260619 16:52:17.325254 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.324764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46884)
0619 16:52:17.324953 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.325052 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 16:52:17.325501 28451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:45755: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab.
Entry for principal kudu/127.14.153.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab.
I20260619 16:52:17.344393 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.61:45755
--webserver_interface=127.14.153.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab
--principal=kudu/127.14.153.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34173,127.14.153.61:45755,127.14.153.60:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:17.374912 28451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } attempt: 1
I20260619 16:52:17.376262 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.375689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46886)
0619 16:52:17.375918 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.376077 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 16:52:17.377411 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.376996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46902)
0619 16:52:17.377154 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.377240 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 16:52:17.377658 28451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:45755: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
I20260619 16:52:17.445938 28451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } attempt: 2
I20260619 16:52:17.447261 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.446664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46914)
0619 16:52:17.446934 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.447088 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 16:52:17.448405 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.447972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46930)
0619 16:52:17.448128 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.448224 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 16:52:17.448679 28451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:45755: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
I20260619 16:52:17.540004 28451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } attempt: 3
I20260619 16:52:17.541390 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.540896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46938)
0619 16:52:17.541151 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.541250 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 16:52:17.542591 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.542182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46948)
0619 16:52:17.542309 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.542414 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 16:52:17.542876 28451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:45755: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
W20260619 16:52:17.679193 28458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:17.679510 28458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:17.679708 28458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:52:17.686141 28451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } attempt: 4
W20260619 16:52:17.687472 28458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260619 16:52:17.687520 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.686902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46962)
0619 16:52:17.687272 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.687385 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
W20260619 16:52:17.687570 28458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:17.687652 28458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:17.687789 28458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:17.687873 28458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:17.688550 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.688187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46966)
0619 16:52:17.688343 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.688430 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 16:52:17.688732 28451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:45755: Network error: Client connection negotiation failed: client connection to 127.14.153.61:45755: connect: Connection refused (error 111)
I20260619 16:52:17.698164 28458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34173,127.14.153.61:45755,127.14.153.60:35649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab
--principal=kudu/127.14.153.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.61:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.153.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:17.699662 28458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:17.700985 28458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:17.713265 28464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.713680 28466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.713984 28463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:17.714349 28458 server_base.cc:1034] running on GCE node
I20260619 16:52:17.715268 28458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:17.716799 28458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:17.718034 28458 hybrid_clock.cc:648] HybridClock initialized: now 1781887937717991 us; error 44 us; skew 500 ppm
I20260619 16:52:17.718403 28458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:17 dist-test-slave-qb36 krb5kdc[28371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:17.721700 28458 init.cc:377] Logged in from keytab as kudu/127.14.153.61@KRBTEST.COM (short username kudu)
I20260619 16:52:17.723338 28458 webserver.cc:460] Webserver started at http://127.14.153.61:35743/ using document root <none> and password file <none>
I20260619 16:52:17.724045 28458 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:17.724187 28458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:17.724504 28458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:17.727576 28458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/data/instance:
uuid: "e240d192aaae46d5a8b9bf117f10b88f"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-qb36"
server_key: "e6e2d0b930949a98940f2c67e2c1d08c"
server_key_iv: "22ac171083940627272452712036f020"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.728396 28458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/wal/instance:
uuid: "e240d192aaae46d5a8b9bf117f10b88f"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-qb36"
server_key: "e6e2d0b930949a98940f2c67e2c1d08c"
server_key_iv: "22ac171083940627272452712036f020"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.733762 28458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 16:52:17.737429 28473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.738803 28458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:52:17.738999 28458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/wal
uuid: "e240d192aaae46d5a8b9bf117f10b88f"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-qb36"
server_key: "e6e2d0b930949a98940f2c67e2c1d08c"
server_key_iv: "22ac171083940627272452712036f020"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.739244 28458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:17.760833 28458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:17.765544 28458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:17.765847 28458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:17.787492 28458 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.61:45755
I20260619 16:52:17.787503 28524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.61:45755 every 8 connection(s)
I20260619 16:52:17.789012 28458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
I20260619 16:52:17.792871 28525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:17.798050 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 28458
I20260619 16:52:17.798261 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-1/wal/instance
I20260619 16:52:17.798823 14948 external_mini_cluster.cc:1428] Setting key ccc8fa931abeb0b2be25064dc8ebfaa6
I20260619 16:52:17.800810 28525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:52:17 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.60@KRBTEST.COM" created.
I20260619 16:52:17.827764 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.805216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34173 (local address 127.0.0.1:40900)
0619 16:52:17.806139 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.806208 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.808136 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.808868 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.808877 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.809448 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:17.810031 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.810048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.811573 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.811576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.812856 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.812872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.813986 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.813990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.814199 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.814935 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:17.814959 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:17.817406 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:17.822124 (+ 4718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.822134 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.822154 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.822980 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.823552 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.823560 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.823564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.823895 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.824669 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:17.824678 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:17.825199 (+ 521us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.825523 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.826135 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":370,"threads_started":1}
I20260619 16:52:17.831876 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.804689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.14.153.62:34173)
0619 16:52:17.804922 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.804929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.806370 (+ 1441us) server_negotiation.cc:407] Connection header received
0619 16:52:17.808433 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.808441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.808540 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.808656 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:17.810235 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.811406 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.813018 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.813775 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.813936 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.817664 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:17.817669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:17.817672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:17.817707 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:17.821895 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.823148 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.823158 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.823162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.823389 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.824068 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.824074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.824077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.824516 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:17.824749 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.831316 (+ 6567us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.831633 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:52:17.833638 28525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:52:17 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
I20260619 16:52:17.849037 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.834545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46976)
0619 16:52:17.834730 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.834754 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.834870 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.835968 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.835973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.836485 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:17.836628 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.836636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.837997 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.838005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.838846 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.838858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.839717 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.839722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.840013 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.840687 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:17.840704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:17.842881 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:17.846382 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.846389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.846393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.846748 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.847198 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.847203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.847206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.847359 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.847995 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:17.847999 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:17.848142 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.848334 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.848629 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":50}
I20260619 16:52:17.849116 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.834787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.14.153.61:45755)
0619 16:52:17.835529 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.835536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.835561 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:52:17.835608 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.835614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.835808 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.835980 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:17.836791 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.837864 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.838983 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.839571 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.839707 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.843105 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:17.843116 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:17.843131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:17.843163 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:17.846248 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.846899 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.846905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.846907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.847011 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.847511 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.847514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.847516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.847862 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:17.848075 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.848484 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.848737 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":588,"thread_start_us":467,"threads_started":1}
I20260619 16:52:17.852042 28525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:52:17.853341 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.852833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40730)
0619 16:52:17.853039 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.853165 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:52:17.853628 28476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab.
Entry for principal kudu/127.14.153.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab.
I20260619 16:52:17.855334 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.854867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40746)
0619 16:52:17.855094 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.855175 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:52:17.855567 28525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:35649: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
I20260619 16:52:17.856722 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.60:35649
--webserver_interface=127.14.153.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab
--principal=kudu/127.14.153.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34173,127.14.153.61:45755,127.14.153.60:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:17.904982 28525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } attempt: 1
I20260619 16:52:17.906505 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.905823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40748)
0619 16:52:17.906099 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.906252 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 16:52:17.908277 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.907881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40758)
0619 16:52:17.908046 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.908132 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:52:17.908545 28525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:35649: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
I20260619 16:52:17.976819 28525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } attempt: 2
I20260619 16:52:17.978219 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.977576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40764)
0619 16:52:17.977853 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.977986 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 16:52:17.979468 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.978961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40768)
0619 16:52:17.979193 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.979299 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:52:17.979751 28525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:35649: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
I20260619 16:52:17.987980 28451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } attempt: 5
Jun 19 16:52:17 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
I20260619 16:52:18.003192 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.988737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46988)
0619 16:52:17.988960 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.988983 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.989122 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.989560 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.989563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.989939 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:17.990120 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.990128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.991549 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.991553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.992412 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.992423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.993625 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.993629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.993756 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.994407 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:17.994424 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:17.996545 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:18.000647 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.000651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.000654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.001079 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.001526 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.001529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.001532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.001671 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.002354 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:18.002358 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:18.002502 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 16:52:18.002623 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:52:18.002849 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:52:18.003547 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.988991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.14.153.61:45755)
0619 16:52:17.989190 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.989196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.989219 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:52:17.989302 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.989307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.989404 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.989516 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:17.990295 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.991421 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.992612 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.993345 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.993634 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.996762 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:17.996770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:17.996775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:17.996817 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.000445 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.001220 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.001226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.001229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.001397 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.001795 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.001800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.001804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.002223 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.002390 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.002792 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.003115 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:52:18.004720 28451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:52:18.005762 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.005368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40770)
0619 16:52:18.005546 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.005623 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 16:52:18.006819 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.006477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40774)
0619 16:52:18.006616 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.006704 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 16:52:18.007164 28451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:35649: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
I20260619 16:52:18.058501 28451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } attempt: 1
I20260619 16:52:18.059794 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.059333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40784)
0619 16:52:18.059531 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.059625 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 16:52:18.060892 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.060535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40796)
0619 16:52:18.060696 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.060778 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 16:52:18.061182 28451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:35649: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
I20260619 16:52:18.071009 28525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } attempt: 3
I20260619 16:52:18.072414 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.071761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40804)
0619 16:52:18.072031 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.072171 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 16:52:18.073830 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.073223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40812)
0619 16:52:18.073518 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.073628 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260619 16:52:18.074080 28525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:35649: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
I20260619 16:52:18.129429 28451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } attempt: 2
I20260619 16:52:18.130628 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.130116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40816)
0619 16:52:18.130331 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.130466 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 16:52:18.131788 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.131462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40830)
0619 16:52:18.131591 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.131669 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 16:52:18.132052 28451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:35649: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
W20260619 16:52:18.198305 28532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:18.198594 28532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:18.198768 28532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:18.206044 28532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:18.206166 28532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:18.206212 28532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:18.206332 28532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:18.206403 28532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:18.217327 28525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } attempt: 4
I20260619 16:52:18.218590 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.218023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40834)
0619 16:52:18.218273 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.218391 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 16:52:18.218469 28532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34173,127.14.153.61:45755,127.14.153.60:35649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab
--principal=kudu/127.14.153.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.60:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.153.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:18.219601 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.219239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40842)
0619 16:52:18.219395 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.219474 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:52:18.219823 28525 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:35649: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
I20260619 16:52:18.220238 28532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:18.221516 28532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:18.233616 28540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:18.233705 28538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:18.233680 28537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:18.235337 28532 server_base.cc:1034] running on GCE node
I20260619 16:52:18.236181 28532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:18.237777 28532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:18.238276 28451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } attempt: 3
I20260619 16:52:18.239070 28532 hybrid_clock.cc:648] HybridClock initialized: now 1781887938238995 us; error 63 us; skew 500 ppm
I20260619 16:52:18.239502 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.238907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40858)
0619 16:52:18.239196 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.239301 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 16:52:18.239519 28532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:52:18.240592 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.240236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40866)
0619 16:52:18.240419 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.240490 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 16:52:18.240887 28451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:35649: Network error: Client connection negotiation failed: client connection to 127.14.153.60:35649: connect: Connection refused (error 111)
Jun 19 16:52:18 dist-test-slave-qb36 krb5kdc[28371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:18.243405 28532 init.cc:377] Logged in from keytab as kudu/127.14.153.60@KRBTEST.COM (short username kudu)
I20260619 16:52:18.245144 28532 webserver.cc:460] Webserver started at http://127.14.153.60:46197/ using document root <none> and password file <none>
I20260619 16:52:18.245939 28532 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:18.246073 28532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:18.246464 28532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:18.249722 28532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/data/instance:
uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50"
format_stamp: "Formatted at 2026-06-19 16:52:18 on dist-test-slave-qb36"
server_key: "e759e1613d09a79ec9222d0a844e7abb"
server_key_iv: "34dfc3edffe3966d0464054bce22e03e"
server_key_version: "encryptionkey@0"
I20260619 16:52:18.250619 28532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/wal/instance:
uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50"
format_stamp: "Formatted at 2026-06-19 16:52:18 on dist-test-slave-qb36"
server_key: "e759e1613d09a79ec9222d0a844e7abb"
server_key_iv: "34dfc3edffe3966d0464054bce22e03e"
server_key_version: "encryptionkey@0"
I20260619 16:52:18.256008 28532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:52:18.259507 28547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:18.260829 28532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:52:18.261041 28532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/wal
uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50"
format_stamp: "Formatted at 2026-06-19 16:52:18 on dist-test-slave-qb36"
server_key: "e759e1613d09a79ec9222d0a844e7abb"
server_key_iv: "34dfc3edffe3966d0464054bce22e03e"
server_key_version: "encryptionkey@0"
I20260619 16:52:18.261251 28532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:18.277334 28532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:18.282112 28532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:18.282435 28532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:18.302385 28532 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.60:35649
I20260619 16:52:18.302425 28598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.60:35649 every 8 connection(s)
I20260619 16:52:18.303857 28532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
I20260619 16:52:18.307633 28599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:18.308909 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 28532
I20260619 16:52:18.309122 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887734944185-14948-0/minicluster-data/master-2/wal/instance
I20260619 16:52:18.309696 14948 external_mini_cluster.cc:1428] Setting key cd73cb4b17238db4e3080720ae645091
I20260619 16:52:18.317778 28599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:52:18 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
Jun 19 16:52:18 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:18.335212 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.311804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.14.153.60:35649)
0619 16:52:18.312686 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.312698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.312740 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:52:18.314345 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.314368 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.314791 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.315230 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.317106 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.318417 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.320487 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.321521 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.321724 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.326233 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.326244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.326265 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.326313 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.330166 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.331100 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.331108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.331121 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.331315 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.331711 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.331718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.331723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.332499 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.332799 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.333243 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.333701 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":423,"threads_started":1}
I20260619 16:52:18.337450 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.320805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.14.153.62:34173)
0619 16:52:18.321062 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.321071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.321926 (+ 855us) server_negotiation.cc:407] Connection header received
0619 16:52:18.322193 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.322198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.322268 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.322375 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.323569 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.324291 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.326081 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.326827 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.326988 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.331145 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.331151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.331157 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.331205 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.334591 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.335397 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.335404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.335407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.335580 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.335982 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.335986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.335989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.336332 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.336524 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.337015 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.337190 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:52:18.337648 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.320827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34173 (local address 127.0.0.1:40916)
0619 16:52:18.321785 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.321834 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:18.322044 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:18.322447 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:18.322454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:18.323144 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:18.323418 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.323426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.324443 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.324450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.325876 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.325893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.327085 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.327094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.327304 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.327984 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:18.328009 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:18.330873 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:18.334726 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.334735 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.334741 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.335246 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.335710 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.335713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.335715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.335858 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.336488 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:18.336495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:18.336680 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 16:52:18.336898 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 16:52:18.337245 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":451,"threads_started":1}
I20260619 16:52:18.339340 28599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:52:18 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
Jun 19 16:52:18 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:18.353325 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.340015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:45755 (local address 127.0.0.1:46998)
0619 16:52:18.340172 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.340191 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:18.340286 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:18.340636 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:18.340639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:18.340949 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:18.341148 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.341159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.342234 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.342242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.343663 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.343679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.344400 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.344406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.344627 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.345321 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:18.345338 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:18.347495 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:18.350786 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.350792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.350797 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.351306 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.351701 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.351706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.351710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.351857 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.352343 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:18.352348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:18.352524 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 16:52:18.352658 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:52:18.352952 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:52:18.353356 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.340126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.14.153.61:45755)
0619 16:52:18.340336 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.340342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.340362 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:52:18.340427 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.340434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.340510 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.340623 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.341349 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.342076 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.343802 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.344244 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.344405 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.347684 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.347688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.347692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.347724 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.350654 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.351440 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.351445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.351449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.351587 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.351972 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.351981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.351983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.352228 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.352392 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.352773 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.352990 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:52:18.354375 28599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:52:18.354578 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.338908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.14.153.62:34173)
0619 16:52:18.339192 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.339197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.339215 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:52:18.339356 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.339364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.339438 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.339535 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.341029 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.341862 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.343860 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.344249 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.344384 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.348864 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.348870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.348873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.348903 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.351756 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.352456 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.352460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.352462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.352588 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.352977 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.352987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.352989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.353240 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.353766 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.354017 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.354276 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
Jun 19 16:52:18 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
Jun 19 16:52:18 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
I20260619 16:52:18.368721 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.355177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40878)
0619 16:52:18.355317 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.355336 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:18.355428 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:18.355707 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:18.355711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:18.356159 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:18.356341 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.356352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.357441 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.357449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.358598 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.358609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.359458 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.359464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.359665 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.360285 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:18.360310 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:18.362688 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:18.366066 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.366072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.366077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.366550 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.366921 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.366924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.366927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.367119 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.367789 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:18.367792 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:18.367931 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 16:52:18.368077 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:52:18.368414 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 16:52:18.369032 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.355235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.14.153.60:35649)
0619 16:52:18.355391 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.355396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.355414 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:52:18.355498 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.355502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.355573 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.355692 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.356489 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.357308 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.358728 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.359305 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.359403 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.362886 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.362895 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.362903 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.362947 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.365937 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.366659 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.366664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.366668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.366803 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.367261 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.367264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.367266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.367664 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.367821 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.368215 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.368624 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":57}
I20260619 16:52:18.372814 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.356251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.14.153.61:45755)
0619 16:52:18.356409 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.356432 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.356451 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:52:18.356601 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.356606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.356674 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.356771 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.358125 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.358898 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.360543 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.360948 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.361040 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.366472 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.366477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.366480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.366514 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.369997 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.370687 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.370693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.370697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.370845 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.371298 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.371309 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.371311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.371599 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.371936 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.372231 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.372477 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:52:18.374068 28599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50: Bootstrap starting.
I20260619 16:52:18.377857 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:52:18.377954 28599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:18.379269 28599 log.cc:824] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:18.382700 28599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50: No bootstrap required, opened a new log
I20260619 16:52:18.395583 28599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } }
I20260619 16:52:18.396088 28599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:18.396235 28599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee4e1caccfe4d9f8331d1cca0cccc50, State: Initialized, Role: FOLLOWER
I20260619 16:52:18.397100 28599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } }
I20260619 16:52:18.401384 28612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.401834 28612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:52:18.402949 28599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:18.405035 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.384186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.14.153.62:34173)
0619 16:52:18.384493 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.384502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.384528 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:52:18.384732 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.384739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.384823 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.384896 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.387359 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.388487 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.390690 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.391360 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.391459 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.395770 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.395777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.395782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.395838 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.400010 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.402070 (+ 2060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.402076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.402080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.402258 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.402808 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.402813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.402816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.403259 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.403496 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.404309 (+ 813us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.404588 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 16:52:18.411372 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.383713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.14.153.61:45755)
0619 16:52:18.383885 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.383891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.390556 (+ 6665us) server_negotiation.cc:407] Connection header received
0619 16:52:18.391132 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.391142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.391253 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.391340 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.394423 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.395880 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.398505 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.399220 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.399330 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.403243 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.403250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.403255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.403304 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.407508 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.408540 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.408548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.408553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.408741 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.409286 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.409293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.409299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.409710 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.409942 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.410557 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.410866 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:52:18.415529 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.385626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.14.153.60:35649)
0619 16:52:18.385865 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.385887 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.397059 (+ 11172us) server_negotiation.cc:407] Connection header received
0619 16:52:18.397556 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.397566 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.397646 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.397710 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.399504 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.400680 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.402983 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.403793 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.403926 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.407899 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.407909 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.407915 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.407967 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.412175 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.413287 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.413292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.413294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.413442 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.413848 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.413852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.413858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.414159 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.414432 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.414738 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.414982 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:52:18.418134 28451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } attempt: 4
W20260619 16:52:18.422493 28623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:52:18.422637 28623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 16:52:18 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
I20260619 16:52:18.419276 28532 master_runner.cc:186] Error getting master registration for 127.14.153.62:34173: OK, instance_id { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" instance_seqno: 1781887937277170 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:52:18.430685 28532 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:52:18.436337 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.418762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40884)
0619 16:52:18.418943 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.418963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:18.419107 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:18.419483 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:18.419488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:18.420035 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:18.420254 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.420270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.421837 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.421845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.423410 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.423426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.425792 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.425800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.426018 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.426730 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:18.426756 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:18.429764 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:18.433933 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.433937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.433940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.434303 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.434746 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.434749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.434751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.434892 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.435568 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:18.435575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:18.435735 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 16:52:18.435839 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:52:18.436063 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 16:52:18.436985 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.418844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.14.153.60:35649)
0619 16:52:18.419002 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.419009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.419049 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:52:18.419267 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.419272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.419350 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.419455 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.420444 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.421643 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.424852 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.425620 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.425776 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.429991 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.429998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.430004 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.430048 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.433789 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.434445 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.434451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.434455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.434636 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.435012 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.435017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.435020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.435437 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.435593 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.436000 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.436251 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:52:18.442945 28451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87: Bootstrap starting.
I20260619 16:52:18.447279 28451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:18.448647 28451 log.cc:824] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:18.452123 28451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87: No bootstrap required, opened a new log
I20260619 16:52:18.463701 28451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } }
I20260619 16:52:18.464169 28451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:18.464301 28451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884ba9b377cd423ebd22d0b028c98d87, State: Initialized, Role: FOLLOWER
I20260619 16:52:18.465018 28451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } }
I20260619 16:52:18.466100 28626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.466471 28626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:52:18.467551 28451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:52:18.479439 28637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:52:18.479643 28637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:52:18.477672 28384 master_runner.cc:186] Error getting master registration for 127.14.153.61:45755: OK, instance_id { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" instance_seqno: 1781887937780143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:52:18.489624 28384 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:52:18.517026 28405 tablet.cc:2369] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:18.519048 28525 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } attempt: 5
I20260619 16:52:18.521504 28626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:18.521811 28626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } }
I20260619 16:52:18.523512 28626 leader_election.cc:288] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e240d192aaae46d5a8b9bf117f10b88f (127.14.153.61:45755), 8ee4e1caccfe4d9f8331d1cca0cccc50 (127.14.153.60:35649)
I20260619 16:52:18.525314 28574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "884ba9b377cd423ebd22d0b028c98d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" is_pre_election: true
I20260619 16:52:18.526047 28574 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 884ba9b377cd423ebd22d0b028c98d87 in term 0.
I20260619 16:52:18.526692 28402 leader_election.cc:302] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 884ba9b377cd423ebd22d0b028c98d87, 8ee4e1caccfe4d9f8331d1cca0cccc50; no voters:
I20260619 16:52:18.527144 28626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:52:18.527295 28626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:52:18.527411 28626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:52:18 dist-test-slave-qb36 krb5kdc[28371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
I20260619 16:52:18.530783 28626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } }
I20260619 16:52:18.532437 28574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "884ba9b377cd423ebd22d0b028c98d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50"
I20260619 16:52:18.532707 28574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:18.534282 28626 leader_election.cc:288] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [CANDIDATE]: Term 1 election: Requested vote from peers e240d192aaae46d5a8b9bf117f10b88f (127.14.153.61:45755), 8ee4e1caccfe4d9f8331d1cca0cccc50 (127.14.153.60:35649)
W20260619 16:52:18.535003 28553 tablet.cc:2369] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:18.536494 28574 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 884ba9b377cd423ebd22d0b028c98d87 in term 1.
I20260619 16:52:18.536964 28402 leader_election.cc:302] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 884ba9b377cd423ebd22d0b028c98d87, 8ee4e1caccfe4d9f8331d1cca0cccc50; no voters:
I20260619 16:52:18.539151 28626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:18.542964 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.519592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:35649 (local address 127.0.0.1:40900)
0619 16:52:18.519844 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:18.519865 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:18.519974 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:18.520720 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:18.520725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:18.521291 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:18.521505 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.521518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.523011 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.523017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.524499 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:18.524516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.526067 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.526072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:18.526314 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.527057 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:18.527083 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:18.530911 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:18.540026 (+ 9115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.540032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.540037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.540593 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.541037 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:18.541042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:18.541046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:18.541203 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:18.541933 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:18.541939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:18.542099 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 16:52:18.542253 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 16:52:18.542568 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 16:52:18.544014 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:18.520094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.14.153.60:35649)
0619 16:52:18.520266 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:18.520272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:18.520295 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:52:18.520350 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:18.520355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:18.520442 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:18.520537 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:18.521679 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.522829 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.524931 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:18.525680 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:18.525814 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:18.531187 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:18.531194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:18.531200 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:18.531242 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:18.539833 (+ 8591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.540738 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.540743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.540747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.540906 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:18.541332 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:18.541337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:18.541346 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:18.541703 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:18.541868 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:18.543353 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0619 16:52:18.543611 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:52:18.550005 28525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f: Bootstrap starting.
I20260619 16:52:18.550835 28626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [term 1 LEADER]: Becoming Leader. State: Replica: 884ba9b377cd423ebd22d0b028c98d87, State: Running, Role: LEADER
I20260619 16:52:18.554589 28525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:18.556061 28525 log.cc:824] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:18.552152 28626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } }
I20260619 16:52:18.559662 28525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f: No bootstrap required, opened a new log
I20260619 16:52:18.571177 28525 raft_consensus.cc:348] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } }
I20260619 16:52:18.571650 28525 raft_consensus.cc:374] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:18.571779 28525 raft_consensus.cc:729] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e240d192aaae46d5a8b9bf117f10b88f, State: Initialized, Role: FOLLOWER
I20260619 16:52:18.572580 28525 consensus_queue.cc:260] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } }
I20260619 16:52:18.579582 28642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.580041 28642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:52:18.581991 28525 sys_catalog.cc:564] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:18.582170 28639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884ba9b377cd423ebd22d0b028c98d87. Latest consensus state: current_term: 1 leader_uuid: "884ba9b377cd423ebd22d0b028c98d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.582656 28639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:18.583364 28500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "884ba9b377cd423ebd22d0b028c98d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e240d192aaae46d5a8b9bf117f10b88f" is_pre_election: true
I20260619 16:52:18.584033 28500 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 884ba9b377cd423ebd22d0b028c98d87 in term 0.
I20260619 16:52:18.585342 28499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "884ba9b377cd423ebd22d0b028c98d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e240d192aaae46d5a8b9bf117f10b88f"
I20260619 16:52:18.585616 28499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:18.589234 28499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 884ba9b377cd423ebd22d0b028c98d87 in term 1.
I20260619 16:52:18.597621 28643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:18.600152 28643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:18.620028 28574 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [term 1 FOLLOWER]: Refusing update from remote peer 884ba9b377cd423ebd22d0b028c98d87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:18.620617 28499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [term 1 FOLLOWER]: Refusing update from remote peer 884ba9b377cd423ebd22d0b028c98d87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:18.621061 28639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:18.621456 28626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:18.643085 28642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884ba9b377cd423ebd22d0b028c98d87. Latest consensus state: current_term: 1 leader_uuid: "884ba9b377cd423ebd22d0b028c98d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.643460 28642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:52:18.645437 28612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884ba9b377cd423ebd22d0b028c98d87. Latest consensus state: current_term: 1 leader_uuid: "884ba9b377cd423ebd22d0b028c98d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.645750 28612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:52:18.648839 28652 mvcc.cc:205] Tried to move back new op lower bound from 7298612996575657984 to 7298612996336271360. Current Snapshot: MvccSnapshot[applied={T|T < 7298612996575657984}]
I20260619 16:52:18.652626 28651 mvcc.cc:205] Tried to move back new op lower bound from 7298612996575657984 to 7298612996336271360. Current Snapshot: MvccSnapshot[applied={T|T < 7298612996575657984}]
I20260619 16:52:18.659603 28612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "884ba9b377cd423ebd22d0b028c98d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.659889 28612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:52:18.661737 28642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "884ba9b377cd423ebd22d0b028c98d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.661988 28642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:52:18.669999 28643 catalog_manager.cc:1347] Generated new cluster ID: 7c427981a493445f8fc454150ada09a2
I20260619 16:52:18.670123 28643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:18.680199 28661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "884ba9b377cd423ebd22d0b028c98d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.680789 28661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:18.681142 28639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "884ba9b377cd423ebd22d0b028c98d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884ba9b377cd423ebd22d0b028c98d87" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34173 } } peers { permanent_uuid: "e240d192aaae46d5a8b9bf117f10b88f" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 45755 } } peers { permanent_uuid: "8ee4e1caccfe4d9f8331d1cca0cccc50" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 35649 } } }
I20260619 16:52:18.681452 28639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:18.711090 28663 catalog_manager.cc:1259] Loaded cluster ID: 7c427981a493445f8fc454150ada09a2
I20260619 16:52:18.711220 28663 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f: loading cluster ID for follower catalog manager: success
W20260619 16:52:18.712517 28663 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260619 16:52:18.712661 28663 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:52:18.723119 28643 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:52:18.725054 28643 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:18.740391 28643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 884ba9b377cd423ebd22d0b028c98d87: Generated new TSK 0
I20260619 16:52:18.741542 28643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:52:19.424636 28623 catalog_manager.cc:1259] Loaded cluster ID: 7c427981a493445f8fc454150ada09a2
I20260619 16:52:19.424743 28623 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50: loading cluster ID for follower catalog manager: success
I20260619 16:52:19.427165 28623 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50: acquiring CA information for follower catalog manager: success
I20260619 16:52:19.428869 28623 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ee4e1caccfe4d9f8331d1cca0cccc50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:52:19.715912 28663 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f: acquiring CA information for follower catalog manager: success
I20260619 16:52:19.717883 28663 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e240d192aaae46d5a8b9bf117f10b88f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:52:19.877250 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 28384
I20260619 16:52:19.878572 28446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:19.989902 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 28384
I20260619 16:52:20.011858 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 28458
I20260619 16:52:20.012985 28520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:20.119719 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 28458
I20260619 16:52:20.139827 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 28532
I20260619 16:52:20.141160 28594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:20.249085 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 28532
2026-06-19T16:52:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4750 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:52:20 dist-test-slave-qb36 krb5kdc[28675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:52:20 dist-test-slave-qb36 krb5kdc[28675](info): set up 2 sockets
Jun 19 16:52:20 dist-test-slave-qb36 krb5kdc[28675](info): commencing operation
krb5kdc: starting...
W20260619 16:52:21.371642 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:52:21 dist-test-slave-qb36 krb5kdc[28675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:52:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:52:21.611608 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35185
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:21.949380 28688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:21.949819 28688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:21.950011 28688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:21.958005 28688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:21.958168 28688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:21.958288 28688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:21.958385 28688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:21.958463 28688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:21.968752 28688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:21.970139 28688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:21.971529 28688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:21.983541 28696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:21.983565 28693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:21.983551 28694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:21.984663 28688 server_base.cc:1034] running on GCE node
I20260619 16:52:21.985481 28688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:21.986938 28688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:21.988173 28688 hybrid_clock.cc:648] HybridClock initialized: now 1781887941988112 us; error 56 us; skew 500 ppm
I20260619 16:52:21.988544 28688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:21 dist-test-slave-qb36 krb5kdc[28675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887941, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:21.991796 28688 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:52:21.993203 28688 webserver.cc:460] Webserver started at http://127.14.153.62:46067/ using document root <none> and password file <none>
I20260619 16:52:21.993944 28688 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:21.994052 28688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:21.994313 28688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:21.997346 28688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "2258fb4ab3b6405aa76a5128d7a9198e"
format_stamp: "Formatted at 2026-06-19 16:52:21 on dist-test-slave-qb36"
server_key: "48fd9ca7646f64f36ab3628041bb8b7e"
server_key_iv: "6fc11c8b607b40ec37d1026adc902850"
server_key_version: "encryptionkey@0"
I20260619 16:52:21.998067 28688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "2258fb4ab3b6405aa76a5128d7a9198e"
format_stamp: "Formatted at 2026-06-19 16:52:21 on dist-test-slave-qb36"
server_key: "48fd9ca7646f64f36ab3628041bb8b7e"
server_key_iv: "6fc11c8b607b40ec37d1026adc902850"
server_key_version: "encryptionkey@0"
I20260619 16:52:22.003736 28688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 16:52:22.007265 28703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:22.008671 28688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:52:22.008862 28688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "2258fb4ab3b6405aa76a5128d7a9198e"
format_stamp: "Formatted at 2026-06-19 16:52:21 on dist-test-slave-qb36"
server_key: "48fd9ca7646f64f36ab3628041bb8b7e"
server_key_iv: "6fc11c8b607b40ec37d1026adc902850"
server_key_version: "encryptionkey@0"
I20260619 16:52:22.009071 28688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:22.041493 28688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:22.046464 28688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:22.046789 28688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:22.064913 28688 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35185
I20260619 16:52:22.064909 28754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35185 every 8 connection(s)
I20260619 16:52:22.066349 28688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:22.070255 28755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:22.075371 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 28688
I20260619 16:52:22.075596 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:52:22.076256 14948 external_mini_cluster.cc:1428] Setting key 62d7b68d4e454ed9409948aa6b91a154
I20260619 16:52:22.081894 28755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e: Bootstrap starting.
I20260619 16:52:22.085781 28755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:22.087114 28755 log.cc:824] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:22.090791 28755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e: No bootstrap required, opened a new log
Jun 19 16:52:22 dist-test-slave-qb36 krb5kdc[28675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:22.101580 28755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258fb4ab3b6405aa76a5128d7a9198e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35185 } }
I20260619 16:52:22.102027 28758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:22.078109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.14.153.62:35185)
0619 16:52:22.078995 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:22.079010 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:22.079101 (+ 91us) server_negotiation.cc:407] Connection header received
0619 16:52:22.080877 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:22.080900 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:22.081300 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:22.081631 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:22.083606 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:22.084950 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:22.086912 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:22.087942 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:22.088158 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:22.093032 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:22.093042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:22.093062 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:22.093123 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:22.097359 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:22.098075 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:22.098086 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:22.098099 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:22.098317 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:22.098687 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:22.098691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:22.098694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:22.099513 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:22.099853 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:22.100141 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 16:52:22.100620 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":421,"threads_started":1}
I20260619 16:52:22.102049 28755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:22.102167 28755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2258fb4ab3b6405aa76a5128d7a9198e, State: Initialized, Role: FOLLOWER
I20260619 16:52:22.102744 28755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258fb4ab3b6405aa76a5128d7a9198e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35185 } }
I20260619 16:52:22.102959 28755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:22.103160 28755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:22.103359 28755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:22.105914 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:52:22.105913 28755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258fb4ab3b6405aa76a5128d7a9198e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35185 } }
I20260619 16:52:22.106454 28755 leader_election.cc:302] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2258fb4ab3b6405aa76a5128d7a9198e; no voters:
I20260619 16:52:22.107110 28755 leader_election.cc:288] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:52:22.107220 28760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:22.109226 28755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:22.109596 28760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [term 1 LEADER]: Becoming Leader. State: Replica: 2258fb4ab3b6405aa76a5128d7a9198e, State: Running, Role: LEADER
I20260619 16:52:22.110240 28760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258fb4ab3b6405aa76a5128d7a9198e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35185 } }
I20260619 16:52:22.113448 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2258fb4ab3b6405aa76a5128d7a9198e. Latest consensus state: current_term: 1 leader_uuid: "2258fb4ab3b6405aa76a5128d7a9198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258fb4ab3b6405aa76a5128d7a9198e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35185 } } }
I20260619 16:52:22.113583 28761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2258fb4ab3b6405aa76a5128d7a9198e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2258fb4ab3b6405aa76a5128d7a9198e" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35185 } } }
I20260619 16:52:22.113982 28761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:22.113986 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:22.114820 28773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:22.119861 28773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:22.128952 28773 catalog_manager.cc:1347] Generated new cluster ID: ec7fcd77500c48db98a9787fb3914f0c
I20260619 16:52:22.129084 28773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:22.149866 28773 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:52:22.151366 28773 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:22.159612 28773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2258fb4ab3b6405aa76a5128d7a9198e: Generated new TSK 0
I20260619 16:52:22.160761 28773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:52:22.219424 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 28688
I20260619 16:52:22.220736 28750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:22.320070 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 28688
2026-06-19T16:52:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2065 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:52:22 dist-test-slave-qb36 krb5kdc[28783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:52:22 dist-test-slave-qb36 krb5kdc[28783](info): set up 2 sockets
Jun 19 16:52:22 dist-test-slave-qb36 krb5kdc[28783](info): commencing operation
krb5kdc: starting...
W20260619 16:52:23.431670 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:52:23 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:52:23.666442 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35693
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:23.997953 28796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:23.998364 28796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:23.998544 28796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:24.005900 28796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:24.006013 28796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:24.006120 28796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:24.006232 28796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:24.006299 28796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:24.016291 28796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:24.017714 28796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:24.019071 28796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:24.031312 28802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:24.031345 28801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:24.031270 28804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:24.032192 28796 server_base.cc:1034] running on GCE node
I20260619 16:52:24.032990 28796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:24.034415 28796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:24.035645 28796 hybrid_clock.cc:648] HybridClock initialized: now 1781887944035591 us; error 32 us; skew 500 ppm
I20260619 16:52:24.036051 28796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:24 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887944, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:24.039379 28796 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:52:24.040956 28796 webserver.cc:460] Webserver started at http://127.14.153.62:36883/ using document root <none> and password file <none>
I20260619 16:52:24.041664 28796 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:24.041807 28796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:24.042116 28796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:24.045149 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "d9d089bd0bf2465e9eb47e6baeac1248"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "adb13a112a8664125b9a020da57c8ebd"
server_key_iv: "4b411e4412a95a4133562194933fbf07"
server_key_version: "encryptionkey@0"
I20260619 16:52:24.045923 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "d9d089bd0bf2465e9eb47e6baeac1248"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "adb13a112a8664125b9a020da57c8ebd"
server_key_iv: "4b411e4412a95a4133562194933fbf07"
server_key_version: "encryptionkey@0"
I20260619 16:52:24.051142 28796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 16:52:24.054574 28811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:24.056005 28796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:52:24.056202 28796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "d9d089bd0bf2465e9eb47e6baeac1248"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "adb13a112a8664125b9a020da57c8ebd"
server_key_iv: "4b411e4412a95a4133562194933fbf07"
server_key_version: "encryptionkey@0"
I20260619 16:52:24.056422 28796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:24.070650 28796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:24.074822 28796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:24.075161 28796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:24.093961 28796 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35693
I20260619 16:52:24.093991 28862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35693 every 8 connection(s)
I20260619 16:52:24.095455 28796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:24.099231 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 28796
I20260619 16:52:24.099407 28863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:24.099452 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:52:24.100291 14948 external_mini_cluster.cc:1428] Setting key 879b103b00ac4e3871b028278f56a497
I20260619 16:52:24.113696 28863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Bootstrap starting.
Jun 19 16:52:24 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:24.117889 28863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:24.119475 28863 log.cc:824] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:24.123050 28863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: No bootstrap required, opened a new log
I20260619 16:52:24.126220 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:24.102402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.14.153.62:35693)
0619 16:52:24.103246 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:24.103258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:24.103297 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:52:24.105620 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:24.105644 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:24.106060 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:24.106414 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:24.108148 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:24.109490 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:24.111445 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:24.112103 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:24.112227 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:24.117232 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:24.117244 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:24.117264 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:24.117321 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:24.121439 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:24.122194 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:24.122204 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:24.122213 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:24.122455 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:24.122820 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:24.122825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:24.122828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:24.123586 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:24.123929 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:24.124233 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 16:52:24.124702 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":640,"thread_start_us":406,"threads_started":1}
I20260619 16:52:24.132539 28863 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:24.132939 28863 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:24.133018 28863 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d089bd0bf2465e9eb47e6baeac1248, State: Initialized, Role: FOLLOWER
I20260619 16:52:24.133574 28863 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:24.133782 28863 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:24.133879 28863 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:24.134006 28863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:24.136402 28863 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:24.136927 28863 leader_election.cc:302] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9d089bd0bf2465e9eb47e6baeac1248; no voters:
I20260619 16:52:24.137545 28863 leader_election.cc:288] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:52:24.137679 28868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:24.138355 28868 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 1 LEADER]: Becoming Leader. State: Replica: d9d089bd0bf2465e9eb47e6baeac1248, State: Running, Role: LEADER
I20260619 16:52:24.138899 28868 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:24.139631 28863 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:24.141731 28869 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } } }
I20260619 16:52:24.141706 28870 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9d089bd0bf2465e9eb47e6baeac1248. Latest consensus state: current_term: 1 leader_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } } }
I20260619 16:52:24.142153 28869 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:24.142153 28870 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:24.144788 28874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:24.149356 28874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:24.158499 28874 catalog_manager.cc:1347] Generated new cluster ID: ea3b4bc77c314db98220f19895c6e15d
I20260619 16:52:24.158696 28874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:24.171801 28874 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:52:24.173215 28874 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:24.179631 28874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Generated new TSK 0
I20260619 16:52:24.180732 28874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:52:24.244938 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35693
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:24.577001 28889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:24.577289 28889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:24.577452 28889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:24.584685 28889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:24.584813 28889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:24.585021 28889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:52:24.594909 28889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:24.596354 28889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:24.597684 28889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:24.611361 28895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:24.611393 28897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:24.611433 28894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:24.612203 28889 server_base.cc:1034] running on GCE node
I20260619 16:52:24.612800 28889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:24.613744 28889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:24.614954 28889 hybrid_clock.cc:648] HybridClock initialized: now 1781887944614911 us; error 32 us; skew 500 ppm
I20260619 16:52:24.615295 28889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:24 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887944, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:24.618614 28889 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:52:24.620155 28889 webserver.cc:460] Webserver started at http://127.14.153.1:36035/ using document root <none> and password file <none>
I20260619 16:52:24.620838 28889 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:24.620959 28889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:24.621289 28889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:24.624281 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "7807891d5621493daac221d4d4b44dff"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "6c2e7cbd183d82ea5c760c17142aead1"
server_key_iv: "1780752c64fed76f729bdbc6a825d889"
server_key_version: "encryptionkey@0"
I20260619 16:52:24.625101 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "7807891d5621493daac221d4d4b44dff"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "6c2e7cbd183d82ea5c760c17142aead1"
server_key_iv: "1780752c64fed76f729bdbc6a825d889"
server_key_version: "encryptionkey@0"
I20260619 16:52:24.630576 28889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:52:24.634151 28904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:24.635599 28889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:52:24.635772 28889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "7807891d5621493daac221d4d4b44dff"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "6c2e7cbd183d82ea5c760c17142aead1"
server_key_iv: "1780752c64fed76f729bdbc6a825d889"
server_key_version: "encryptionkey@0"
I20260619 16:52:24.635967 28889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:24.652065 28889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:24.656656 28889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:24.656952 28889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:24.657946 28889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:24.659361 28889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:24.659433 28889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:24.659582 28889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:24.659663 28889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:24.693699 28889 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:41061
I20260619 16:52:24.693786 29016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:41061 every 8 connection(s)
I20260619 16:52:24.694995 28889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:52:24.698132 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 28889
I20260619 16:52:24.698347 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:52:24.699258 14948 external_mini_cluster.cc:1428] Setting key 460456973217a8c0765c263d3e00c0fb
Jun 19 16:52:24 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887944, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:24.717080 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:24.697561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:53239 (local address 127.14.153.62:35693)
0619 16:52:24.698029 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:24.698034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:24.699180 (+ 1146us) server_negotiation.cc:407] Connection header received
0619 16:52:24.701539 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:24.701544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:24.701641 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:24.701760 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:24.703321 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:24.704030 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:24.705583 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:24.705986 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:24.706113 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:24.710062 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:24.710071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:24.710079 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:24.710123 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:24.713616 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:24.714387 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:24.714396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:24.714402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:24.714553 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:24.714984 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:24.714992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:24.714998 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:24.715402 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:24.715567 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:24.716309 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 16:52:24.716575 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":259,"threads_started":1}
I20260619 16:52:24.717790 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:24.698163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:53239)
0619 16:52:24.698982 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:24.699067 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:24.701248 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:24.702005 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:24.702014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:24.702526 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:24.703153 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:24.703172 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:24.704192 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:24.704197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:24.705429 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:24.705440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:24.706226 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:24.706233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:24.706470 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:24.707162 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:24.707216 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:24.709843 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:24.713817 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:24.713824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:24.713838 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:24.714251 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:24.714655 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:24.714658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:24.714661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:24.714855 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:24.715520 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:24.715527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:24.715887 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 16:52:24.716124 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 16:52:24.716564 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":333,"threads_started":1}
I20260619 16:52:24.719664 29017 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35693
I20260619 16:52:24.719977 29017 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:24.720777 29017 heartbeater.cc:511] Master 127.14.153.62:35693 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:52:24.723204 28828 ts_manager.cc:194] Registered new tserver with Master: 7807891d5621493daac221d4d4b44dff (127.14.153.1:41061)
I20260619 16:52:24.724699 28828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:53239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:52:24.754583 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35693
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:25.083518 29023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:25.083806 29023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:25.083957 29023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:25.091982 29023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:25.092094 29023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:25.092295 29023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:52:25.102329 29023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:25.103679 29023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:25.104970 29023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:25.118237 29028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:25.118305 29029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:25.118327 29031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:25.119289 29023 server_base.cc:1034] running on GCE node
I20260619 16:52:25.119918 29023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:25.120891 29023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:25.122104 29023 hybrid_clock.cc:648] HybridClock initialized: now 1781887945122040 us; error 63 us; skew 500 ppm
I20260619 16:52:25.122414 29023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:25 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887945, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:25.125598 29023 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:52:25.127148 29023 webserver.cc:460] Webserver started at http://127.14.153.2:42085/ using document root <none> and password file <none>
I20260619 16:52:25.127847 29023 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:25.127951 29023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:25.128196 29023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:25.131083 29023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "ff45e17bc9234608832517a470d62569"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e409d0eb7363f5299af9691698c6683"
server_key_iv: "c315bdeda5a93f2a4039403d46e69fe5"
server_key_version: "encryptionkey@0"
I20260619 16:52:25.131831 29023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "ff45e17bc9234608832517a470d62569"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e409d0eb7363f5299af9691698c6683"
server_key_iv: "c315bdeda5a93f2a4039403d46e69fe5"
server_key_version: "encryptionkey@0"
I20260619 16:52:25.137058 29023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:52:25.140686 29038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:25.142158 29023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:52:25.142310 29023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "ff45e17bc9234608832517a470d62569"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e409d0eb7363f5299af9691698c6683"
server_key_iv: "c315bdeda5a93f2a4039403d46e69fe5"
server_key_version: "encryptionkey@0"
I20260619 16:52:25.142496 29023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:25.172988 29023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:25.177260 29023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:25.177527 29023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:25.178458 29023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:25.179929 29023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:25.180004 29023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:25.180137 29023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:25.180199 29023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:25.213984 29023 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:41313
I20260619 16:52:25.214028 29150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:41313 every 8 connection(s)
I20260619 16:52:25.215406 29023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:52:25.217690 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 29023
I20260619 16:52:25.217893 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:52:25.218456 14948 external_mini_cluster.cc:1428] Setting key 146ab7249d1c1578b385bcbb43a64ca9
Jun 19 16:52:25 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887945, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:25.241236 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.218388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:39109 (local address 127.14.153.62:35693)
0619 16:52:25.218883 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:25.218890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:25.220138 (+ 1248us) server_negotiation.cc:407] Connection header received
0619 16:52:25.222995 (+ 2857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:25.223000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:25.223137 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:25.223240 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:25.225388 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.226146 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.228673 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.229382 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.229526 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.233081 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:25.233090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:25.233097 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:25.233142 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:25.236853 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:25.238481 (+ 1628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:25.238487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:25.238491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:25.238678 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:25.239240 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:25.239245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:25.239248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:25.239586 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:25.239733 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:25.240597 (+ 864us) server_negotiation.cc:299] Negotiation successful
0619 16:52:25.240839 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":250,"threads_started":1}
I20260619 16:52:25.242375 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.219247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:39109)
0619 16:52:25.219992 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:25.220031 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:25.222672 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:25.223518 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:25.223533 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:25.224353 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:25.225201 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.225228 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.226299 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.226308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.228505 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.228523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.229614 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.229619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.229839 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.230563 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:25.230587 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:25.232841 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:25.236994 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:25.237003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:25.237023 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:25.238296 (+ 1273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:25.238823 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:25.238828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:25.238831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:25.239118 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:25.239706 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:25.239715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:25.240235 (+ 520us) client_negotiation.cc:769] Sending connection context
0619 16:52:25.240501 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 16:52:25.241137 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":320,"threads_started":1}
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:52:25.243921 29151 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35693
I20260619 16:52:25.244288 29151 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:25.245085 29151 heartbeater.cc:511] Master 127.14.153.62:35693 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
I20260619 16:52:25.246668 28828 ts_manager.cc:194] Registered new tserver with Master: ff45e17bc9234608832517a470d62569 (127.14.153.2:41313)
I20260619 16:52:25.247615 28828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:39109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:52:25.274361 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35693
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:25.610162 29157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:25.610452 29157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:25.610546 29157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:25.618037 29157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:25.618157 29157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:25.618286 29157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:52:25.628440 29157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:25.629786 29157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:25.631173 29157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:25.644778 29162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:25.644784 29163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:25.644907 29165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:25.645454 29157 server_base.cc:1034] running on GCE node
I20260619 16:52:25.646131 29157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:25.647496 29157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:25.648723 29157 hybrid_clock.cc:648] HybridClock initialized: now 1781887945648666 us; error 52 us; skew 500 ppm
I20260619 16:52:25.649065 29157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:25 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887945, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:25.652441 29157 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:52:25.654067 29157 webserver.cc:460] Webserver started at http://127.14.153.3:46281/ using document root <none> and password file <none>
I20260619 16:52:25.654775 29157 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:25.654917 29157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:25.655289 29157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:25.658371 29157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e315941de4d7a13424432d46fe9ed38"
server_key_iv: "784937e906c77f45dc7f4b050894ea56"
server_key_version: "encryptionkey@0"
I20260619 16:52:25.659224 29157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e315941de4d7a13424432d46fe9ed38"
server_key_iv: "784937e906c77f45dc7f4b050894ea56"
server_key_version: "encryptionkey@0"
I20260619 16:52:25.664582 29157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 16:52:25.668107 29172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:25.669467 29157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:52:25.669644 29157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e315941de4d7a13424432d46fe9ed38"
server_key_iv: "784937e906c77f45dc7f4b050894ea56"
server_key_version: "encryptionkey@0"
I20260619 16:52:25.669842 29157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:25.689062 29157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:25.693542 29157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:25.693868 29157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:25.695134 29157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:25.696920 29157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:25.697014 29157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:25.697183 29157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:25.697259 29157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:25.727916 29017 heartbeater.cc:503] Master 127.14.153.62:35693 was elected leader, sending a full tablet report...
I20260619 16:52:25.731930 29157 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:32771
I20260619 16:52:25.731976 29284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:32771 every 8 connection(s)
I20260619 16:52:25.733210 29157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:25.738435 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 29157
I20260619 16:52:25.738621 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:52:25.739142 14948 external_mini_cluster.cc:1428] Setting key 141b736bf4675039686e18fe45c3c712
Jun 19 16:52:25 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887945, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:25.755450 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.735811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:34529 (local address 127.14.153.62:35693)
0619 16:52:25.735989 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:25.735994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:25.737126 (+ 1132us) server_negotiation.cc:407] Connection header received
0619 16:52:25.739395 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:25.739401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:25.739509 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:25.739632 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:25.741915 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.742875 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.744736 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.745167 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.745274 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.749237 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:25.749242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:25.749247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:25.749282 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:25.751851 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:25.752611 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:25.752617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:25.752620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:25.752789 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:25.753465 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:25.753469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:25.753478 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:25.753870 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:25.754029 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:25.754936 (+ 907us) server_negotiation.cc:299] Negotiation successful
0619 16:52:25.755208 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:52:25.756605 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.736247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:34529)
0619 16:52:25.736959 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:25.736999 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:25.738991 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:25.739966 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:25.739982 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:25.740828 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:25.741724 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.741752 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.743021 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.743065 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.744604 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.744618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.745353 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.745378 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.745619 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.746349 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:25.746378 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:25.749007 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:25.751999 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:25.752008 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:25.752028 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:25.752470 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:25.752920 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:25.752924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:25.752926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:25.753335 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:25.754014 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:25.754021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:25.754484 (+ 463us) client_negotiation.cc:769] Sending connection context
0619 16:52:25.754816 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 16:52:25.755374 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":273,"threads_started":1}
I20260619 16:52:25.757989 29285 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35693
I20260619 16:52:25.758296 29285 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:25.759092 29285 heartbeater.cc:511] Master 127.14.153.62:35693 requested a full tablet report, sending...
I20260619 16:52:25.760691 28828 ts_manager.cc:194] Registered new tserver with Master: 7aaf50a8c6e342c2a29fa5a996e6f22b (127.14.153.3:32771)
I20260619 16:52:25.761643 28828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:34529
I20260619 16:52:25.767577 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:52:25.789420 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.771884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.14.153.62:35693)
0619 16:52:25.772121 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:25.772128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:25.772410 (+ 282us) server_negotiation.cc:407] Connection header received
0619 16:52:25.772556 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:25.772559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:25.772629 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:25.772709 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:25.774189 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.774933 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.776339 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.776759 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.776855 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.779375 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:25.779380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:25.779384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:25.779421 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:25.784852 (+ 5431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:25.786275 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:25.786284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:25.786290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:25.786481 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:25.787238 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:25.787244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:25.787247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:25.787720 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:25.788008 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:25.788790 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 16:52:25.789048 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:52:25.795236 28828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:52:25.805115 28828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:52:25.839895 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.817668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:41313 (local address 127.0.0.1:34782)
0619 16:52:25.828844 (+ 11176us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:25.828916 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:25.829206 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:25.831925 (+ 2719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:25.831932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:25.831970 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:25.832514 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.832529 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.834430 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.834435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.836848 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.836862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.838717 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.838723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.838931 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.838988 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 16:52:25.839155 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 16:52:25.839392 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10777,"mutex_wait_us":65,"thread_start_us":14248,"threads_started":1}
I20260619 16:52:25.840466 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.818239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.14.153.3:32771)
0619 16:52:25.831773 (+ 13534us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:25.831786 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:25.832351 (+ 565us) server_negotiation.cc:407] Connection header received
0619 16:52:25.832530 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:25.832538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:25.832763 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:25.832979 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:25.834057 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.835707 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.837830 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.839260 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.839403 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.839674 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:25.839925 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 16:52:25.840129 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13294,"thread_start_us":13097,"threads_started":1}
I20260619 16:52:25.840551 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.818045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:32771 (local address 127.0.0.1:48968)
0619 16:52:25.832209 (+ 14164us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:25.832233 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:25.832368 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:25.833273 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:25.833277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:25.833303 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:25.833815 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.833834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.835828 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.835835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.837628 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.837641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.839523 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.839532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.839725 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.839759 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:52:25.839877 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 16:52:25.840009 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14024,"thread_start_us":11417,"threads_started":1}
I20260619 16:52:25.841749 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.817930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.14.153.2:41313)
0619 16:52:25.831227 (+ 13297us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:25.831241 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:25.831293 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:52:25.831416 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:25.831424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:25.831681 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:25.831903 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:25.832697 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.834272 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.837016 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.838526 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.838664 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.838868 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:25.840469 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0619 16:52:25.841426 (+ 957us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13061,"thread_start_us":12882,"threads_started":1}
I20260619 16:52:25.843245 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.833283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41061 (local address 127.0.0.1:58354)
0619 16:52:25.833930 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:25.833949 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:25.834138 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:25.834974 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:25.834978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:25.835002 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:25.835452 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.835462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.837315 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.837323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.840433 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:25.840446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.842466 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.842470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:25.842590 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.842610 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 16:52:25.842714 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:52:25.842835 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":4,"thread_start_us":296,"threads_started":1}
I20260619 16:52:25.843268 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:25.832670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.14.153.1:41061)
0619 16:52:25.833299 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:25.833307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:25.834035 (+ 728us) server_negotiation.cc:407] Connection header received
0619 16:52:25.834328 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:25.834337 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:25.834551 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:25.834704 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:25.835643 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.837151 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.840616 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:25.842090 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:25.842171 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:25.842319 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:25.842761 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 16:52:25.842910 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":288,"threads_started":1}
I20260619 16:52:25.844595 29220 tablet_service.cc:1467] Processing CreateTablet for tablet 87c475e025b94451b80d551a17d5eac6 (DEFAULT_TABLE table=test-table [id=f4436d0d68ec4a1d8878695da39fe21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:25.846520 28952 tablet_service.cc:1467] Processing CreateTablet for tablet 87c475e025b94451b80d551a17d5eac6 (DEFAULT_TABLE table=test-table [id=f4436d0d68ec4a1d8878695da39fe21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:25.847470 29220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c475e025b94451b80d551a17d5eac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:25.847805 28952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c475e025b94451b80d551a17d5eac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:25.847793 29086 tablet_service.cc:1467] Processing CreateTablet for tablet 87c475e025b94451b80d551a17d5eac6 (DEFAULT_TABLE table=test-table [id=f4436d0d68ec4a1d8878695da39fe21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:25.848851 29086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c475e025b94451b80d551a17d5eac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:25.860216 29303 tablet_bootstrap.cc:492] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: Bootstrap starting.
I20260619 16:52:25.862244 29305 tablet_bootstrap.cc:492] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Bootstrap starting.
I20260619 16:52:25.865540 29303 tablet_bootstrap.cc:654] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:25.865955 29305 tablet_bootstrap.cc:654] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:25.866973 29303 log.cc:824] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:25.867166 29305 log.cc:824] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:25.867923 29304 tablet_bootstrap.cc:492] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: Bootstrap starting.
I20260619 16:52:25.871192 29304 tablet_bootstrap.cc:654] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:25.872285 29304 log.cc:824] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:25.876497 29304 tablet_bootstrap.cc:492] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: No bootstrap required, opened a new log
I20260619 16:52:25.876605 29305 tablet_bootstrap.cc:492] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: No bootstrap required, opened a new log
I20260619 16:52:25.876608 29303 tablet_bootstrap.cc:492] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: No bootstrap required, opened a new log
I20260619 16:52:25.876816 29304 ts_tablet_manager.cc:1397] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260619 16:52:25.876886 29305 ts_tablet_manager.cc:1397] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260619 16:52:25.876883 29303 ts_tablet_manager.cc:1397] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20260619 16:52:25.885291 29304 raft_consensus.cc:348] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:25.885736 29304 raft_consensus.cc:374] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:25.885841 29304 raft_consensus.cc:729] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7807891d5621493daac221d4d4b44dff, State: Initialized, Role: FOLLOWER
I20260619 16:52:25.886535 29304 consensus_queue.cc:260] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:25.888100 29305 raft_consensus.cc:348] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:25.888101 29303 raft_consensus.cc:348] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:25.888499 29304 ts_tablet_manager.cc:1428] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 16:52:25.888557 29303 raft_consensus.cc:374] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:25.888557 29305 raft_consensus.cc:374] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:25.888654 29303 raft_consensus.cc:729] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff45e17bc9234608832517a470d62569, State: Initialized, Role: FOLLOWER
I20260619 16:52:25.888654 29305 raft_consensus.cc:729] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aaf50a8c6e342c2a29fa5a996e6f22b, State: Initialized, Role: FOLLOWER
I20260619 16:52:25.889273 29305 consensus_queue.cc:260] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:25.889312 29303 consensus_queue.cc:260] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:25.890883 29151 heartbeater.cc:503] Master 127.14.153.62:35693 was elected leader, sending a full tablet report...
I20260619 16:52:25.891198 29305 ts_tablet_manager.cc:1428] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260619 16:52:25.891453 29303 ts_tablet_manager.cc:1428] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260619 16:52:25.892060 29285 heartbeater.cc:503] Master 127.14.153.62:35693 was elected leader, sending a full tablet report...
W20260619 16:52:25.947438 29018 tablet.cc:2369] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:52:25.967531 29152 tablet.cc:2369] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:52:25.984881 29286 tablet.cc:2369] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:26.078169 29310 raft_consensus.cc:482] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:26.078516 29310 raft_consensus.cc:504] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:26.080363 29310 leader_election.cc:288] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff45e17bc9234608832517a470d62569 (127.14.153.2:41313), 7807891d5621493daac221d4d4b44dff (127.14.153.1:41061)
I20260619 16:52:26.087445 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:26.080709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:41313 (local address 127.14.153.3:45881)
0619 16:52:26.080967 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:26.080988 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:26.081223 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:26.081802 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:26.081808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:26.081846 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:26.082062 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:26.082072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.083404 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.083410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:26.085367 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:26.085383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.086776 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.086780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:26.086914 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:26.086939 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:52:26.087012 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:52:26.087178 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":31}
I20260619 16:52:26.087492 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:26.080799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:45881 (local address 127.14.153.2:41313)
0619 16:52:26.080990 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:26.080996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:26.081081 (+ 85us) server_negotiation.cc:407] Connection header received
0619 16:52:26.081419 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:26.081427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:26.081536 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:26.081659 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:26.082218 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.083231 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.085546 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.086606 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.086799 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:26.086880 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:26.087090 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 16:52:26.087223 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:52:26.088274 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:26.081444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:41501 (local address 127.14.153.1:41061)
0619 16:52:26.081696 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:26.081703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:26.082045 (+ 342us) server_negotiation.cc:407] Connection header received
0619 16:52:26.082281 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:26.082288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:26.082366 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:26.082468 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:26.082949 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.083805 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.085996 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.087305 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.087457 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:26.087539 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:26.087940 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 16:52:26.088049 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 16:52:26.088398 29312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:26.081171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41061 (local address 127.14.153.3:41501)
0619 16:52:26.081916 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:26.081932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:26.082054 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:26.082495 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:26.082498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:26.082519 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:26.082788 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:26.082795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.083948 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.083953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:26.085847 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:26.085857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.087515 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.087533 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:26.087751 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:26.087792 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:52:26.087874 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:52:26.088021 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":1,"thread_start_us":391,"threads_started":1}
I20260619 16:52:26.088780 29106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c475e025b94451b80d551a17d5eac6" candidate_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45e17bc9234608832517a470d62569" is_pre_election: true
I20260619 16:52:26.089370 29106 raft_consensus.cc:2454] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aaf50a8c6e342c2a29fa5a996e6f22b in term 0.
I20260619 16:52:26.089458 28972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c475e025b94451b80d551a17d5eac6" candidate_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7807891d5621493daac221d4d4b44dff" is_pre_election: true
I20260619 16:52:26.090056 28972 raft_consensus.cc:2454] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aaf50a8c6e342c2a29fa5a996e6f22b in term 0.
I20260619 16:52:26.090111 29173 leader_election.cc:302] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aaf50a8c6e342c2a29fa5a996e6f22b, ff45e17bc9234608832517a470d62569; no voters:
I20260619 16:52:26.090552 29310 raft_consensus.cc:2781] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:52:26.090741 29310 raft_consensus.cc:482] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:52:26.090873 29310 raft_consensus.cc:3037] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:26.093182 29310 raft_consensus.cc:504] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:26.093883 29310 leader_election.cc:288] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [CANDIDATE]: Term 1 election: Requested vote from peers ff45e17bc9234608832517a470d62569 (127.14.153.2:41313), 7807891d5621493daac221d4d4b44dff (127.14.153.1:41061)
I20260619 16:52:26.094415 29106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c475e025b94451b80d551a17d5eac6" candidate_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45e17bc9234608832517a470d62569"
I20260619 16:52:26.094483 28972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c475e025b94451b80d551a17d5eac6" candidate_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7807891d5621493daac221d4d4b44dff"
I20260619 16:52:26.094671 29106 raft_consensus.cc:3037] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:26.094679 28972 raft_consensus.cc:3037] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:26.097971 28972 raft_consensus.cc:2454] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aaf50a8c6e342c2a29fa5a996e6f22b in term 1.
I20260619 16:52:26.097986 29106 raft_consensus.cc:2454] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aaf50a8c6e342c2a29fa5a996e6f22b in term 1.
I20260619 16:52:26.098408 29173 leader_election.cc:302] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aaf50a8c6e342c2a29fa5a996e6f22b, ff45e17bc9234608832517a470d62569; no voters:
I20260619 16:52:26.098706 29310 raft_consensus.cc:2781] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:26.099247 29310 raft_consensus.cc:686] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 1 LEADER]: Becoming Leader. State: Replica: 7aaf50a8c6e342c2a29fa5a996e6f22b, State: Running, Role: LEADER
I20260619 16:52:26.099790 29310 consensus_queue.cc:237] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:26.105171 28827 catalog_manager.cc:5495] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7aaf50a8c6e342c2a29fa5a996e6f22b (127.14.153.3). New cstate: current_term: 1 leader_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } health_report { overall_health: UNKNOWN } } }
I20260619 16:52:26.140540 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:26.135126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.14.153.3:32771)
0619 16:52:26.135305 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:26.135309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:26.135385 (+ 76us) server_negotiation.cc:407] Connection header received
0619 16:52:26.135562 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:26.135566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:26.135643 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:26.135789 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:52:26.136105 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.136840 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.138332 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.138723 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.138826 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:26.139371 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:52:26.139951 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:52:26.140120 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:26.140167 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 16:52:26.140235 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:52:26.148846 29106 raft_consensus.cc:1261] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [term 1 FOLLOWER]: Refusing update from remote peer 7aaf50a8c6e342c2a29fa5a996e6f22b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:26.148941 28972 raft_consensus.cc:1261] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [term 1 FOLLOWER]: Refusing update from remote peer 7aaf50a8c6e342c2a29fa5a996e6f22b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:26.149811 29310 consensus_queue.cc:1035] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:26.150130 29313 consensus_queue.cc:1035] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:26.164409 29317 mvcc.cc:205] Tried to move back new op lower bound from 7298613027415187456 to 7298613027229077504. Current Snapshot: MvccSnapshot[applied={T|T < 7298613027415187456}]
I20260619 16:52:26.165916 29318 mvcc.cc:205] Tried to move back new op lower bound from 7298613027415187456 to 7298613027229077504. Current Snapshot: MvccSnapshot[applied={T|T < 7298613027415187456}]
I20260619 16:52:26.177906 28826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:26.178215 28826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:26.181996 28826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Sending DeleteTablet for 3 replicas of tablet 87c475e025b94451b80d551a17d5eac6
I20260619 16:52:26.183328 29220 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c475e025b94451b80d551a17d5eac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:26 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:48968
I20260619 16:52:26.183431 28952 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c475e025b94451b80d551a17d5eac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:26 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:58354
I20260619 16:52:26.184083 29086 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c475e025b94451b80d551a17d5eac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:26 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:34782
I20260619 16:52:26.184358 29326 tablet_replica.cc:330] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: stopping tablet replica
I20260619 16:52:26.184922 29326 raft_consensus.cc:2229] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:52:26.185825 29326 raft_consensus.cc:2258] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:26.186426 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 28889
I20260619 16:52:26.188813 29326 ts_tablet_manager.cc:1905] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:26.196559 29328 tablet_replica.cc:330] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: stopping tablet replica
I20260619 16:52:26.197058 29328 raft_consensus.cc:2229] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:26.197606 29328 raft_consensus.cc:2258] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:26.198930 29326 ts_tablet_manager.cc:1918] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:26.199184 29326 log.cc:1196] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/87c475e025b94451b80d551a17d5eac6
I20260619 16:52:26.199892 29328 ts_tablet_manager.cc:1905] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:26.200778 29326 ts_tablet_manager.cc:1939] T 87c475e025b94451b80d551a17d5eac6 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Deleting consensus metadata
I20260619 16:52:26.202651 28814 catalog_manager.cc:4841] TS 7aaf50a8c6e342c2a29fa5a996e6f22b (127.14.153.3:32771): tablet 87c475e025b94451b80d551a17d5eac6 (table test-table [id=f4436d0d68ec4a1d8878695da39fe21a]) successfully deleted
I20260619 16:52:26.206099 29328 ts_tablet_manager.cc:1918] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:26.206262 29328 log.cc:1196] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/87c475e025b94451b80d551a17d5eac6
I20260619 16:52:26.206830 29328 ts_tablet_manager.cc:1939] T 87c475e025b94451b80d551a17d5eac6 P ff45e17bc9234608832517a470d62569: Deleting consensus metadata
I20260619 16:52:26.207476 29327 tablet_replica.cc:330] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: stopping tablet replica
I20260619 16:52:26.207942 29327 raft_consensus.cc:2229] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:26.208802 29327 raft_consensus.cc:2258] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:26.208794 28812 catalog_manager.cc:4841] TS ff45e17bc9234608832517a470d62569 (127.14.153.2:41313): tablet 87c475e025b94451b80d551a17d5eac6 (table test-table [id=f4436d0d68ec4a1d8878695da39fe21a]) successfully deleted
I20260619 16:52:26.210579 29327 ts_tablet_manager.cc:1905] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:26.217499 29327 ts_tablet_manager.cc:1918] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:26.217667 29327 log.cc:1196] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/87c475e025b94451b80d551a17d5eac6
Jun 19 16:52:26 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:52:26.218207 29327 ts_tablet_manager.cc:1939] T 87c475e025b94451b80d551a17d5eac6 P 7807891d5621493daac221d4d4b44dff: Deleting consensus metadata
I20260619 16:52:26.220280 28812 catalog_manager.cc:4841] TS 7807891d5621493daac221d4d4b44dff (127.14.153.1:41061): tablet 87c475e025b94451b80d551a17d5eac6 (table test-table [id=f4436d0d68ec4a1d8878695da39fe21a]) successfully deleted
I20260619 16:52:26.225688 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:26.187706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.14.153.1:41061)
0619 16:52:26.187908 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:26.187915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:26.199552 (+ 11637us) server_negotiation.cc:407] Connection header received
0619 16:52:26.199777 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:26.199785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:26.199880 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:26.200016 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:26.209318 (+ 9302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.210490 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.212653 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.213328 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.213488 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:26.219493 (+ 6005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:26.219505 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:26.219523 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:26.219572 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:26.222733 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:26.223582 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:26.223589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:26.223592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:26.223722 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:26.224127 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:26.224131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:26.224133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:26.224490 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:26.224813 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:26.225106 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 16:52:26.225334 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:52:26.226426 29012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:26.318603 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 28889
I20260619 16:52:26.337457 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 29023
Jun 19 16:52:26 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:52:26.354735 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:26.338294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.14.153.2:41313)
0619 16:52:26.338526 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:26.338533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:26.338556 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:52:26.338706 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:26.338714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:26.338800 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:26.338911 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:26.340289 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.341487 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.342781 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.343555 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.343683 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:26.348688 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:26.348704 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:26.348722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:26.348768 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:26.351883 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:26.352748 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:26.352767 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:26.352770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:26.352897 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:26.353290 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:26.353293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:26.353295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:26.353664 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:26.353900 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:26.354257 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 16:52:26.354428 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:52:26.355453 29146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:26.445990 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 29023
I20260619 16:52:26.469326 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 29157
Jun 19 16:52:26 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:52:26.489305 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:26.470170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.14.153.3:32771)
0619 16:52:26.470349 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:26.470353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:26.470388 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:52:26.470581 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:26.470585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:26.470655 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:26.470747 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:26.472540 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.473425 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.475335 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:26.475899 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:26.476000 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:26.481505 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:26.481523 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:26.481551 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:26.481623 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:26.486102 (+ 4479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:26.486855 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:26.486865 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:26.486868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:26.487111 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:26.487515 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:26.487520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:26.487523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:26.488027 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:26.488363 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:26.488614 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 16:52:26.488871 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:52:26.490108 29280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:26.612520 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 29157
I20260619 16:52:26.632301 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 28796
I20260619 16:52:26.633447 28858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:26.744045 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 28796
I20260619 16:52:26.765285 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35693
--webserver_interface=127.14.153.62
--webserver_port=36883
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:27.102329 29334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:27.102639 29334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:27.102731 29334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:27.110178 29334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:27.110304 29334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:27.110351 29334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:27.110409 29334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:27.110455 29334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:27.120546 29334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=36883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:27.121881 29334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:27.123246 29334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:27.146739 29340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:27.146816 29341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:27.146844 29343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:27.148039 29334 server_base.cc:1034] running on GCE node
I20260619 16:52:27.148952 29334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:27.150614 29334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:27.151846 29334 hybrid_clock.cc:648] HybridClock initialized: now 1781887947151780 us; error 49 us; skew 500 ppm
I20260619 16:52:27.152297 29334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:27 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887947, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:27.156493 29334 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:52:27.158293 29334 webserver.cc:460] Webserver started at http://127.14.153.62:36883/ using document root <none> and password file <none>
I20260619 16:52:27.159178 29334 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:27.159353 29334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:27.165731 29334 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:52:27.169116 29350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:27.170653 29334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:52:27.170862 29334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "d9d089bd0bf2465e9eb47e6baeac1248"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "adb13a112a8664125b9a020da57c8ebd"
server_key_iv: "4b411e4412a95a4133562194933fbf07"
server_key_version: "encryptionkey@0"
I20260619 16:52:27.172039 29334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:27.209483 29334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:27.214232 29334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:27.214526 29334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:27.233973 29334 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35693
I20260619 16:52:27.233997 29401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35693 every 8 connection(s)
I20260619 16:52:27.235651 29334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:27.239984 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 29334
I20260619 16:52:27.240790 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:41061
--local_ip_for_outbound_sockets=127.14.153.1
--tserver_master_addrs=127.14.153.62:35693
--webserver_port=36035
--webserver_interface=127.14.153.1
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:27.243248 29402 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:52:27.247798 29402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Bootstrap starting.
I20260619 16:52:27.268481 29402 log.cc:824] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:27.281374 29402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:27.281949 29402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Bootstrap complete.
I20260619 16:52:27.291424 29402 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:27.292369 29402 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d089bd0bf2465e9eb47e6baeac1248, State: Initialized, Role: FOLLOWER
I20260619 16:52:27.293006 29402 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:27.293272 29402 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:27.293403 29402 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:27.293579 29402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:52:27.297580 29402 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:27.298117 29402 leader_election.cc:302] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9d089bd0bf2465e9eb47e6baeac1248; no voters:
I20260619 16:52:27.298768 29402 leader_election.cc:288] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:52:27.298908 29406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:52:27.300266 29406 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 2 LEADER]: Becoming Leader. State: Replica: d9d089bd0bf2465e9eb47e6baeac1248, State: Running, Role: LEADER
I20260619 16:52:27.300823 29406 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:27.301219 29402 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:27.303504 29407 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } } }
I20260619 16:52:27.303975 29407 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:27.308436 29412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:27.308354 29408 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9d089bd0bf2465e9eb47e6baeac1248. Latest consensus state: current_term: 2 leader_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } } }
I20260619 16:52:27.308734 29408 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:27.318041 29412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:27.319593 29412 catalog_manager.cc:1259] Loaded cluster ID: ea3b4bc77c314db98220f19895c6e15d
I20260619 16:52:27.319802 29412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:27.323083 29412 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 16:52:27.342439 29412 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 16:52:27.364271 29412 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:27.366166 29412 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Loaded TSK: 0
I20260619 16:52:27.367777 29412 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:52:27.607105 29404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:27.607462 29404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:27.607613 29404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:27.615159 29404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:27.615306 29404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:27.615489 29404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:52:27.625746 29404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:41061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=36035
--tserver_master_addrs=127.14.153.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:27.627204 29404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:27.628576 29404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:27.644335 29429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:27.644317 29430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:27.645239 29432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:27.648576 29404 server_base.cc:1034] running on GCE node
I20260619 16:52:27.649307 29404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:27.651147 29404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:27.652395 29404 hybrid_clock.cc:648] HybridClock initialized: now 1781887947652325 us; error 62 us; skew 500 ppm
I20260619 16:52:27.652861 29404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:27 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887947, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:27.661039 29404 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:52:27.663002 29404 webserver.cc:460] Webserver started at http://127.14.153.1:36035/ using document root <none> and password file <none>
I20260619 16:52:27.664005 29404 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:27.664157 29404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:27.671519 29404 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.003s
I20260619 16:52:27.675266 29439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:27.676885 29404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:52:27.677078 29404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "7807891d5621493daac221d4d4b44dff"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "6c2e7cbd183d82ea5c760c17142aead1"
server_key_iv: "1780752c64fed76f729bdbc6a825d889"
server_key_version: "encryptionkey@0"
I20260619 16:52:27.678102 29404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:27.707196 29404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:27.711345 29404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:27.711623 29404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:27.712603 29404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:27.713979 29404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:27.714097 29404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:27.714227 29404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:27.714311 29404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:27.748122 29404 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:41061
I20260619 16:52:27.748234 29551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:41061 every 8 connection(s)
I20260619 16:52:27.749642 29404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:52:27.755580 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 29404
I20260619 16:52:27.756203 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:41313
--local_ip_for_outbound_sockets=127.14.153.2
--tserver_master_addrs=127.14.153.62:35693
--webserver_port=42085
--webserver_interface=127.14.153.2
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:52:27 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887947, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:27.774391 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:27.752556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:40401 (local address 127.14.153.62:35693)
0619 16:52:27.753181 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:27.753193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:27.753753 (+ 560us) server_negotiation.cc:407] Connection header received
0619 16:52:27.756062 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:27.756075 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:27.756582 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:27.756955 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:27.758526 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:27.759899 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:27.761483 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:27.762148 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:27.762356 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:27.766417 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:27.766425 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:27.766440 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:27.766480 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:27.769585 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:27.770700 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:27.770707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:27.770713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:27.770872 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:27.771425 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:27.771428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:27.771430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:27.771901 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:27.772057 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:27.773073 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0619 16:52:27.773426 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":290,"threads_started":1}
I20260619 16:52:27.775000 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:27.752744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:40401)
0619 16:52:27.753565 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:27.753611 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:27.755511 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:27.757096 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:27.757109 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:27.757742 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:27.758285 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:27.758305 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:27.760076 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:27.760080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:27.761315 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:27.761327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:27.762393 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:27.762400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:27.762673 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:27.763412 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:27.763435 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:27.766132 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:27.769740 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:27.769753 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:27.769774 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:27.770569 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:27.770989 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:27.770997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:27.771003 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:27.771295 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:27.772048 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:27.772060 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:27.772545 (+ 485us) client_negotiation.cc:769] Sending connection context
0619 16:52:27.772887 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 16:52:27.773562 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":348,"threads_started":1}
I20260619 16:52:27.778417 29552 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35693
I20260619 16:52:27.778791 29552 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:27.779858 29552 heartbeater.cc:511] Master 127.14.153.62:35693 requested a full tablet report, sending...
I20260619 16:52:27.782951 29367 ts_manager.cc:194] Registered new tserver with Master: 7807891d5621493daac221d4d4b44dff (127.14.153.1:41061)
I20260619 16:52:27.785254 29367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:40401
W20260619 16:52:28.098505 29556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:28.098804 29556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:28.098901 29556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:28.106510 29556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:28.106635 29556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:28.106755 29556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:52:28.116765 29556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:41313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=42085
--tserver_master_addrs=127.14.153.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:28.118110 29556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:28.119403 29556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:28.133080 29561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:28.133111 29564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:28.133035 29562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:28.133896 29556 server_base.cc:1034] running on GCE node
I20260619 16:52:28.134490 29556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:28.135476 29556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:28.136718 29556 hybrid_clock.cc:648] HybridClock initialized: now 1781887948136650 us; error 47 us; skew 500 ppm
I20260619 16:52:28.137089 29556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:28 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887948, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:28.140463 29556 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:52:28.141907 29556 webserver.cc:460] Webserver started at http://127.14.153.2:42085/ using document root <none> and password file <none>
I20260619 16:52:28.142594 29556 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:28.142745 29556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:28.148083 29556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:52:28.150810 29571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:28.152293 29556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:52:28.152482 29556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "ff45e17bc9234608832517a470d62569"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e409d0eb7363f5299af9691698c6683"
server_key_iv: "c315bdeda5a93f2a4039403d46e69fe5"
server_key_version: "encryptionkey@0"
I20260619 16:52:28.153510 29556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:28.209964 29556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:28.214536 29556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:28.214825 29556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:28.215876 29556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:28.217257 29556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:28.217384 29556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:28.217500 29556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:28.217584 29556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:28.252959 29556 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:41313
I20260619 16:52:28.253011 29683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:41313 every 8 connection(s)
I20260619 16:52:28.254432 29556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:52:28.263451 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 29556
I20260619 16:52:28.264211 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:32771
--local_ip_for_outbound_sockets=127.14.153.3
--tserver_master_addrs=127.14.153.62:35693
--webserver_port=46281
--webserver_interface=127.14.153.3
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:52:28 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887948, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:28.277180 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.257058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:54775 (local address 127.14.153.62:35693)
0619 16:52:28.257243 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.257249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.258490 (+ 1241us) server_negotiation.cc:407] Connection header received
0619 16:52:28.260344 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.260350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.260455 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.260565 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:28.262090 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.262794 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.265240 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.265661 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.265775 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.271234 (+ 5459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:28.271243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:28.271250 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:28.271293 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:28.274019 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:28.274729 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:28.274735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:28.274743 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:28.274872 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:28.275332 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:28.275337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:28.275374 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:28.275745 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:28.275902 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.276592 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.276825 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:52:28.278183 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.257577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:54775)
0619 16:52:28.258327 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:28.258371 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:28.260075 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:28.260804 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:28.260812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:28.261368 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:28.261924 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.261941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.262952 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.262959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.265086 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.265104 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.265898 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.265907 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.266227 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.266946 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:28.266980 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:28.270937 (+ 3957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:28.274162 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:28.274171 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:28.274188 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:28.274603 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:28.274980 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:28.274985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:28.274989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:28.275219 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:28.275869 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:28.275877 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:28.276224 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 16:52:28.276540 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 16:52:28.277075 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":281,"threads_started":1}
I20260619 16:52:28.279729 29684 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35693
I20260619 16:52:28.280074 29684 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:28.280989 29684 heartbeater.cc:511] Master 127.14.153.62:35693 requested a full tablet report, sending...
I20260619 16:52:28.282785 29367 ts_manager.cc:194] Registered new tserver with Master: ff45e17bc9234608832517a470d62569 (127.14.153.2:41313)
I20260619 16:52:28.284159 29367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:54775
W20260619 16:52:28.594224 29687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:28.594522 29687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:28.594617 29687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:28.602052 29687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:28.602171 29687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:28.602298 29687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:52:28.612448 29687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:32771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=46281
--tserver_master_addrs=127.14.153.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:28.613933 29687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:28.615332 29687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:28.628922 29692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:28.628991 29695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:28.629004 29693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:28.629742 29687 server_base.cc:1034] running on GCE node
I20260619 16:52:28.630398 29687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:28.631465 29687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:28.632679 29687 hybrid_clock.cc:648] HybridClock initialized: now 1781887948632628 us; error 48 us; skew 500 ppm
I20260619 16:52:28.633023 29687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:28 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887948, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:28.636358 29687 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:52:28.637944 29687 webserver.cc:460] Webserver started at http://127.14.153.3:46281/ using document root <none> and password file <none>
I20260619 16:52:28.638638 29687 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:28.638856 29687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:28.644711 29687 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 16:52:28.647527 29702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:28.648928 29687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:52:28.649101 29687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e315941de4d7a13424432d46fe9ed38"
server_key_iv: "784937e906c77f45dc7f4b050894ea56"
server_key_version: "encryptionkey@0"
I20260619 16:52:28.650084 29687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:28.675395 29687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:28.680076 29687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:28.680369 29687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:28.681401 29687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:28.682950 29687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:28.683084 29687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:28.683230 29687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:28.683319 29687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:28.716780 29687 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:32771
I20260619 16:52:28.716863 29814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:32771 every 8 connection(s)
I20260619 16:52:28.718282 29687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:28.719519 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 29687
Jun 19 16:52:28 dist-test-slave-qb36 krb5kdc[28783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887948, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:28.735697 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.720616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.14.153.62:35693)
0619 16:52:28.720862 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.720869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.721248 (+ 379us) server_negotiation.cc:407] Connection header received
0619 16:52:28.721443 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.721447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.721542 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.721656 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:28.723586 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.724723 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.726015 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.726623 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.726746 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.729490 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:28.729499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:28.729504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:28.729554 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:28.732817 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:28.733635 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:28.733641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:28.733644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:28.733794 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:28.734142 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:28.734149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:28.734156 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:28.734473 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:28.734678 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.735122 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.735384 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:52:28.740549 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.721657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:33463 (local address 127.14.153.62:35693)
0619 16:52:28.722116 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.722120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.722759 (+ 639us) server_negotiation.cc:407] Connection header received
0619 16:52:28.725126 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.725132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.725216 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.725330 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:28.727214 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.727916 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.729487 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.729987 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.730138 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.734300 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:28.734304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:28.734307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:28.734338 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:28.737504 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:28.738291 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:28.738301 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:28.738303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:28.738404 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:28.738808 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:28.738813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:28.738816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:28.739126 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:28.739272 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.739944 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.740136 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":270,"threads_started":1}
I20260619 16:52:28.741606 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.721825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:33463)
0619 16:52:28.722608 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:28.722649 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:28.724886 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:28.725581 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:28.725590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:28.726190 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:28.726992 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.727012 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.728064 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.728070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.729336 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.729348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.730206 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.730229 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.730558 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.731189 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:28.731221 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:28.734060 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:28.737637 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:28.737645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:28.737663 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:28.738175 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:28.738505 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:28.738510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:28.738514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:28.738688 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:28.739267 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:28.739275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:28.739596 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 16:52:28.739855 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 16:52:28.740379 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":335,"threads_started":1}
I20260619 16:52:28.742870 29815 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35693
I20260619 16:52:28.743204 29815 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:28.744016 29815 heartbeater.cc:511] Master 127.14.153.62:35693 requested a full tablet report, sending...
I20260619 16:52:28.745302 29367 ts_manager.cc:194] Registered new tserver with Master: 7aaf50a8c6e342c2a29fa5a996e6f22b (127.14.153.3:32771)
I20260619 16:52:28.746088 29367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:33463
I20260619 16:52:28.750325 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:52:28.757371 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.751246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.14.153.62:35693)
0619 16:52:28.751435 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.751441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.751960 (+ 519us) server_negotiation.cc:407] Connection header received
0619 16:52:28.752133 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.752138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.752321 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.752475 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:52:28.752969 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.753671 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.755263 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.755693 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.755784 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.756327 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:52:28.756808 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:52:28.756918 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.757049 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.757150 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:52:28.758425 29367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:52:28.765877 29367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:52:28.790103 29825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.777756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.14.153.3:32771)
0619 16:52:28.778578 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.778589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.779194 (+ 605us) server_negotiation.cc:407] Connection header received
0619 16:52:28.779533 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.779543 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.779810 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.779983 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:28.781216 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.783466 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.787110 (+ 3644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.788601 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.788695 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.788963 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.789571 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.789744 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":376,"threads_started":1}
I20260619 16:52:28.790134 29827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.777449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:32771 (local address 127.0.0.1:48998)
0619 16:52:28.778970 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:28.779012 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:28.779317 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:28.780376 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:28.780384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:28.780416 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:28.780952 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.780974 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.783651 (+ 2677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.783656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.786866 (+ 3210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.786895 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.789180 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.789189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.789347 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.789382 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:52:28.789527 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:52:28.789723 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"thread_start_us":1099,"threads_started":1}
I20260619 16:52:28.794221 29750 tablet_service.cc:1467] Processing CreateTablet for tablet 6caeaadd03f34a998ed805f42f4c5f33 (DEFAULT_TABLE table=test-table [id=a7e9ef406f0f45329f2126ea195a1ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:28.795586 29828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.777532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:41313 (local address 127.0.0.1:34800)
0619 16:52:28.779063 (+ 1531us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:28.779081 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:28.779234 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:28.788681 (+ 9447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:28.788686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:28.788713 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:28.789068 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.789080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.791597 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.791603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.793380 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.793392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.794685 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.794691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.794859 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.794901 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:52:28.795006 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:52:28.795170 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"mutex_wait_us":1,"thread_start_us":1264,"threads_started":1}
I20260619 16:52:28.795595 29826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.777982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.14.153.2:41313)
0619 16:52:28.787992 (+ 10010us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.788002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.788039 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:52:28.788147 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.788155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.788425 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.789243 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:28.789472 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.791435 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.793536 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.794435 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.794604 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.794825 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.795101 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.795277 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9835,"thread_start_us":9763,"threads_started":1}
I20260619 16:52:28.795851 29750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caeaadd03f34a998ed805f42f4c5f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:28.797312 29552 heartbeater.cc:503] Master 127.14.153.62:35693 was elected leader, sending a full tablet report...
I20260619 16:52:28.798468 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.779454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41061 (local address 127.0.0.1:58366)
0619 16:52:28.788456 (+ 9002us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:28.788481 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:28.788621 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:28.789866 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:28.789871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:28.789896 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:28.790296 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.790314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.792783 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.792789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.794788 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.794804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.797654 (+ 2850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.797665 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.797861 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.797894 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:52:28.798006 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:52:28.798149 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8853,"thread_start_us":8706,"threads_started":1}
I20260619 16:52:28.799243 29830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.780043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.14.153.1:41061)
0619 16:52:28.789234 (+ 9191us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.789242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.789268 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:52:28.789368 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.789376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.789625 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.790503 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:28.790767 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.792640 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.794927 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.796866 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.797010 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.797215 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.798722 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.798884 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9029,"thread_start_us":8871,"threads_started":1}
I20260619 16:52:28.799851 29619 tablet_service.cc:1467] Processing CreateTablet for tablet 6caeaadd03f34a998ed805f42f4c5f33 (DEFAULT_TABLE table=test-table [id=a7e9ef406f0f45329f2126ea195a1ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:28.801018 29619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caeaadd03f34a998ed805f42f4c5f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:28.806883 29487 tablet_service.cc:1467] Processing CreateTablet for tablet 6caeaadd03f34a998ed805f42f4c5f33 (DEFAULT_TABLE table=test-table [id=a7e9ef406f0f45329f2126ea195a1ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:28.808391 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caeaadd03f34a998ed805f42f4c5f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:28.810876 29831 tablet_bootstrap.cc:492] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Bootstrap starting.
I20260619 16:52:28.814002 29831 tablet_bootstrap.cc:654] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:28.815145 29831 log.cc:824] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:28.817137 29833 tablet_bootstrap.cc:492] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: Bootstrap starting.
I20260619 16:52:28.818454 29831 tablet_bootstrap.cc:492] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: No bootstrap required, opened a new log
I20260619 16:52:28.818744 29831 ts_tablet_manager.cc:1397] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 16:52:28.821066 29833 tablet_bootstrap.cc:654] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:28.822330 29833 log.cc:824] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:28.824373 29834 tablet_bootstrap.cc:492] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: Bootstrap starting.
I20260619 16:52:28.827675 29834 tablet_bootstrap.cc:654] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:28.828763 29834 log.cc:824] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:28.830116 29833 tablet_bootstrap.cc:492] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: No bootstrap required, opened a new log
I20260619 16:52:28.830422 29833 ts_tablet_manager.cc:1397] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260619 16:52:28.830490 29831 raft_consensus.cc:348] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:28.830963 29831 raft_consensus.cc:374] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:28.831097 29831 raft_consensus.cc:729] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aaf50a8c6e342c2a29fa5a996e6f22b, State: Initialized, Role: FOLLOWER
I20260619 16:52:28.831811 29831 consensus_queue.cc:260] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:28.832950 29815 heartbeater.cc:503] Master 127.14.153.62:35693 was elected leader, sending a full tablet report...
I20260619 16:52:28.833554 29834 tablet_bootstrap.cc:492] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: No bootstrap required, opened a new log
I20260619 16:52:28.833607 29831 ts_tablet_manager.cc:1428] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260619 16:52:28.833817 29834 ts_tablet_manager.cc:1397] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260619 16:52:28.841053 29833 raft_consensus.cc:348] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:28.841521 29833 raft_consensus.cc:374] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:28.841631 29833 raft_consensus.cc:729] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff45e17bc9234608832517a470d62569, State: Initialized, Role: FOLLOWER
I20260619 16:52:28.842073 29834 raft_consensus.cc:348] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:28.842291 29833 consensus_queue.cc:260] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:28.842504 29834 raft_consensus.cc:374] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:28.842607 29834 raft_consensus.cc:729] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7807891d5621493daac221d4d4b44dff, State: Initialized, Role: FOLLOWER
I20260619 16:52:28.843321 29834 consensus_queue.cc:260] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:28.843580 29684 heartbeater.cc:503] Master 127.14.153.62:35693 was elected leader, sending a full tablet report...
I20260619 16:52:28.845034 29834 ts_tablet_manager.cc:1428] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260619 16:52:28.845330 29833 ts_tablet_manager.cc:1428] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: Time spent starting tablet: real 0.015s user 0.003s sys 0.010s
I20260619 16:52:28.860515 29839 raft_consensus.cc:482] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:28.860821 29839 raft_consensus.cc:504] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:28.862519 29839 leader_election.cc:288] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aaf50a8c6e342c2a29fa5a996e6f22b (127.14.153.3:32771), ff45e17bc9234608832517a470d62569 (127.14.153.2:41313)
I20260619 16:52:28.869465 29826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.863140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:52843 (local address 127.14.153.2:41313)
0619 16:52:28.863387 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.863393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.864072 (+ 679us) server_negotiation.cc:407] Connection header received
0619 16:52:28.864219 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.864223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.864314 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.864406 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:28.865064 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.865773 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.867728 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.868522 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.868709 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.868807 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.869052 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.869189 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:52:28.869529 29825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.862879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:54551 (local address 127.14.153.3:32771)
0619 16:52:28.863122 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.863132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.863898 (+ 766us) server_negotiation.cc:407] Connection header received
0619 16:52:28.864090 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.864096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.864183 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.864276 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:28.864950 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.865757 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.867768 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.868529 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.868703 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.868787 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.869151 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.869291 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:52:28.869716 29841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.862809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:32771 (local address 127.14.153.1:54551)
0619 16:52:28.863760 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:28.863784 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:28.863951 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:28.864315 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:28.864319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:28.864347 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:28.864750 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.864763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.865886 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.865891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.867596 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.867607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.868809 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.868815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.868946 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.868969 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:52:28.869116 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 16:52:28.869243 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":59,"thread_start_us":835,"threads_started":1}
I20260619 16:52:28.869779 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.862907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:41313 (local address 127.14.153.1:52843)
0619 16:52:28.863951 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:28.863965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:28.864101 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:28.864480 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:28.864483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:28.864514 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:28.864909 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.864921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.865932 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.865935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.867574 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:28.867589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.868762 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.868768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:28.868905 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.868928 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:52:28.869001 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:52:28.869191 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":46,"thread_start_us":430,"threads_started":1}
I20260619 16:52:28.870790 29639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caeaadd03f34a998ed805f42f4c5f33" candidate_uuid: "7807891d5621493daac221d4d4b44dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45e17bc9234608832517a470d62569" is_pre_election: true
I20260619 16:52:28.870792 29770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caeaadd03f34a998ed805f42f4c5f33" candidate_uuid: "7807891d5621493daac221d4d4b44dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" is_pre_election: true
I20260619 16:52:28.871316 29639 raft_consensus.cc:2454] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7807891d5621493daac221d4d4b44dff in term 0.
I20260619 16:52:28.871335 29770 raft_consensus.cc:2454] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7807891d5621493daac221d4d4b44dff in term 0.
I20260619 16:52:28.871974 29440 leader_election.cc:302] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7807891d5621493daac221d4d4b44dff, ff45e17bc9234608832517a470d62569; no voters:
I20260619 16:52:28.872372 29839 raft_consensus.cc:2781] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:52:28.872520 29839 raft_consensus.cc:482] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:52:28.872627 29839 raft_consensus.cc:3037] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:28.874732 29839 raft_consensus.cc:504] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:28.875418 29839 leader_election.cc:288] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [CANDIDATE]: Term 1 election: Requested vote from peers 7aaf50a8c6e342c2a29fa5a996e6f22b (127.14.153.3:32771), ff45e17bc9234608832517a470d62569 (127.14.153.2:41313)
I20260619 16:52:28.875742 29770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caeaadd03f34a998ed805f42f4c5f33" candidate_uuid: "7807891d5621493daac221d4d4b44dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b"
I20260619 16:52:28.875871 29639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caeaadd03f34a998ed805f42f4c5f33" candidate_uuid: "7807891d5621493daac221d4d4b44dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45e17bc9234608832517a470d62569"
I20260619 16:52:28.875944 29770 raft_consensus.cc:3037] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:28.876127 29639 raft_consensus.cc:3037] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:28.878772 29639 raft_consensus.cc:2454] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7807891d5621493daac221d4d4b44dff in term 1.
I20260619 16:52:28.878760 29770 raft_consensus.cc:2454] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7807891d5621493daac221d4d4b44dff in term 1.
I20260619 16:52:28.879190 29442 leader_election.cc:302] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7807891d5621493daac221d4d4b44dff, 7aaf50a8c6e342c2a29fa5a996e6f22b; no voters:
I20260619 16:52:28.879526 29839 raft_consensus.cc:2781] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:28.879995 29839 raft_consensus.cc:686] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 1 LEADER]: Becoming Leader. State: Replica: 7807891d5621493daac221d4d4b44dff, State: Running, Role: LEADER
I20260619 16:52:28.880481 29839 consensus_queue.cc:237] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } }
I20260619 16:52:28.885381 29366 catalog_manager.cc:5495] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff reported cstate change: term changed from 0 to 1, leader changed from <none> to 7807891d5621493daac221d4d4b44dff (127.14.153.1). New cstate: current_term: 1 leader_uuid: "7807891d5621493daac221d4d4b44dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7807891d5621493daac221d4d4b44dff" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41061 } health_report { overall_health: HEALTHY } } }
I20260619 16:52:28.915321 29830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.910254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.14.153.1:41061)
0619 16:52:28.910461 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.910466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.910487 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:52:28.910664 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.910668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.910755 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.910880 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:52:28.911253 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.911975 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.913323 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.913710 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.913811 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.914234 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:52:28.914743 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:52:28.914857 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.914965 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.915050 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:52:28.923743 29770 raft_consensus.cc:1261] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 1 FOLLOWER]: Refusing update from remote peer 7807891d5621493daac221d4d4b44dff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:28.923738 29639 raft_consensus.cc:1261] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [term 1 FOLLOWER]: Refusing update from remote peer 7807891d5621493daac221d4d4b44dff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:28.924741 29842 consensus_queue.cc:1035] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 32771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:28.925098 29839 consensus_queue.cc:1035] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff45e17bc9234608832517a470d62569" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:28.938360 29845 mvcc.cc:205] Tried to move back new op lower bound from 7298613038780420096 to 7298613038618345472. Current Snapshot: MvccSnapshot[applied={T|T < 7298613038780420096}]
I20260619 16:52:28.952689 29366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:28.953025 29366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:28.956949 29366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Sending DeleteTablet for 3 replicas of tablet 6caeaadd03f34a998ed805f42f4c5f33
I20260619 16:52:28.958403 29487 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caeaadd03f34a998ed805f42f4c5f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:28 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:58366
I20260619 16:52:28.958393 29750 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caeaadd03f34a998ed805f42f4c5f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:28 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:48998
I20260619 16:52:28.958565 29619 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caeaadd03f34a998ed805f42f4c5f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:28 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:34800
I20260619 16:52:28.959414 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 29404
I20260619 16:52:28.970011 29857 tablet_replica.cc:330] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: stopping tablet replica
I20260619 16:52:28.970810 29857 raft_consensus.cc:2229] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:28.971838 29857 raft_consensus.cc:2258] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:28.972453 29856 tablet_replica.cc:330] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: stopping tablet replica
I20260619 16:52:28.973026 29856 raft_consensus.cc:2229] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:52:28.973851 29856 raft_consensus.cc:2258] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:28.976770 29856 ts_tablet_manager.cc:1905] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:28.977567 29857 ts_tablet_manager.cc:1905] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:28.986028 29856 ts_tablet_manager.cc:1918] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:28.986250 29856 log.cc:1196] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/6caeaadd03f34a998ed805f42f4c5f33
I20260619 16:52:28.986524 29830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:28.960667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.14.153.1:41061)
0619 16:52:28.960874 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:28.960883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:28.960905 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:52:28.960975 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:28.960983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:28.961082 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:28.961215 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:28.963889 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.965556 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.968321 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:28.969384 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:28.969532 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:28.976812 (+ 7280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:28.976827 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:28.976845 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:28.976970 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:28.981471 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:28.982638 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:28.982652 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:28.982656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:28.982864 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:28.983416 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:28.983421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:28.983423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:28.983905 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:28.984315 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:28.985571 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0619 16:52:28.985835 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:52:28.986847 29856 ts_tablet_manager.cc:1939] T 6caeaadd03f34a998ed805f42f4c5f33 P 7807891d5621493daac221d4d4b44dff: Deleting consensus metadata
I20260619 16:52:28.987701 29547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:28.987869 29855 tablet_replica.cc:330] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: stopping tablet replica
I20260619 16:52:28.988323 29855 raft_consensus.cc:2229] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:28.988946 29855 raft_consensus.cc:2258] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:28.991639 29857 ts_tablet_manager.cc:1918] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:28.991892 29857 log.cc:1196] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/6caeaadd03f34a998ed805f42f4c5f33
I20260619 16:52:28.992154 29855 ts_tablet_manager.cc:1905] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:28.992513 29857 ts_tablet_manager.cc:1939] T 6caeaadd03f34a998ed805f42f4c5f33 P ff45e17bc9234608832517a470d62569: Deleting consensus metadata
I20260619 16:52:28.994343 29351 catalog_manager.cc:4841] TS ff45e17bc9234608832517a470d62569 (127.14.153.2:41313): tablet 6caeaadd03f34a998ed805f42f4c5f33 (table test-table [id=a7e9ef406f0f45329f2126ea195a1ec6]) successfully deleted
I20260619 16:52:28.997143 29855 ts_tablet_manager.cc:1918] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:28.997323 29855 log.cc:1196] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/6caeaadd03f34a998ed805f42f4c5f33
I20260619 16:52:28.997766 29855 ts_tablet_manager.cc:1939] T 6caeaadd03f34a998ed805f42f4c5f33 P 7aaf50a8c6e342c2a29fa5a996e6f22b: Deleting consensus metadata
I20260619 16:52:28.999241 29353 catalog_manager.cc:4841] TS 7aaf50a8c6e342c2a29fa5a996e6f22b (127.14.153.3:32771): tablet 6caeaadd03f34a998ed805f42f4c5f33 (table test-table [id=a7e9ef406f0f45329f2126ea195a1ec6]) successfully deleted
I20260619 16:52:29.092319 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 29404
I20260619 16:52:29.092561 29351 catalog_manager.cc:4841] TS 7807891d5621493daac221d4d4b44dff (127.14.153.1:41061): tablet 6caeaadd03f34a998ed805f42f4c5f33 (table test-table [id=a7e9ef406f0f45329f2126ea195a1ec6]) successfully deleted
I20260619 16:52:29.110409 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 29556
I20260619 16:52:29.124279 29826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:29.111247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.14.153.2:41313)
0619 16:52:29.111415 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:29.111420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:29.111454 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:52:29.111619 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:29.111623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:29.111698 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:29.111788 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:29.113200 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:29.113965 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:29.115427 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:29.115856 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:29.115971 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:29.118572 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:29.118587 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:29.118604 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:29.118645 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:29.121485 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:29.122193 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:29.122200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:29.122203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:29.122354 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:29.122691 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:29.122695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:29.122697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:29.123226 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:29.123448 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:29.123776 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 16:52:29.123956 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:52:29.125046 29679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:29.214627 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 29556
I20260619 16:52:29.232050 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 29687
I20260619 16:52:29.248003 29825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:29.232866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.14.153.3:32771)
0619 16:52:29.233074 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:29.233079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:29.233099 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:52:29.233347 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:29.233353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:29.233440 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:29.233584 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:29.234842 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:29.235989 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:29.237342 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:29.237974 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:29.238094 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:29.240995 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:29.241015 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:29.241037 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:29.241102 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:29.245049 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:29.245814 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:29.245824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:29.245828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:29.245992 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:29.246356 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:29.246361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:29.246365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:29.246807 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:29.247129 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:29.247395 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 16:52:29.247628 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 16:52:29.248713 29810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:29.343461 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 29687
I20260619 16:52:29.360270 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 29334
I20260619 16:52:29.361456 29397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:29.474967 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 29334
I20260619 16:52:29.496944 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35693
--webserver_interface=127.14.153.62
--webserver_port=36883
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:29.832543 29862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:29.832860 29862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:29.832958 29862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:29.840416 29862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:29.840533 29862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:29.840579 29862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:29.840636 29862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:29.840679 29862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:29.850803 29862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=36883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:29.852214 29862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:29.853495 29862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:29.865756 29867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:29.865676 29868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:29.865725 29870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:29.866526 29862 server_base.cc:1034] running on GCE node
I20260619 16:52:29.867394 29862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:29.868851 29862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:29.870049 29862 hybrid_clock.cc:648] HybridClock initialized: now 1781887949869967 us; error 68 us; skew 500 ppm
I20260619 16:52:29.870445 29862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:29 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887949, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:29.873788 29862 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:52:29.875432 29862 webserver.cc:460] Webserver started at http://127.14.153.62:36883/ using document root <none> and password file <none>
I20260619 16:52:29.876165 29862 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:29.876307 29862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:29.882059 29862 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:52:29.885121 29877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:29.886528 29862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:52:29.886715 29862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "d9d089bd0bf2465e9eb47e6baeac1248"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "adb13a112a8664125b9a020da57c8ebd"
server_key_iv: "4b411e4412a95a4133562194933fbf07"
server_key_version: "encryptionkey@0"
I20260619 16:52:29.887773 29862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:29.918386 29862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:29.923132 29862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:29.923424 29862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:29.941431 29862 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35693
I20260619 16:52:29.941444 29928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35693 every 8 connection(s)
I20260619 16:52:29.942991 29862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:29.949749 29929 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:52:29.950493 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 29862
I20260619 16:52:29.951557 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:41061
--local_ip_for_outbound_sockets=127.14.153.1
--tserver_master_addrs=127.14.153.62:35693
--webserver_port=36035
--webserver_interface=127.14.153.1
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:29.954933 29929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Bootstrap starting.
I20260619 16:52:29.977192 29929 log.cc:824] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:29.994333 29929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:29.994930 29929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248: Bootstrap complete.
I20260619 16:52:30.008080 29929 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:30.009359 29929 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d089bd0bf2465e9eb47e6baeac1248, State: Initialized, Role: FOLLOWER
I20260619 16:52:30.010090 29929 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:30.010347 29929 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:30.010445 29929 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:30.010622 29929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:52:30.016388 29929 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:30.017100 29929 leader_election.cc:302] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9d089bd0bf2465e9eb47e6baeac1248; no voters:
I20260619 16:52:30.017897 29929 leader_election.cc:288] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:52:30.018076 29933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:52:30.020161 29929 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:30.021713 29933 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [term 3 LEADER]: Becoming Leader. State: Replica: d9d089bd0bf2465e9eb47e6baeac1248, State: Running, Role: LEADER
I20260619 16:52:30.022382 29933 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } }
I20260619 16:52:30.032981 29934 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } } }
I20260619 16:52:30.033494 29934 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:30.035535 29935 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9d089bd0bf2465e9eb47e6baeac1248. Latest consensus state: current_term: 3 leader_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d089bd0bf2465e9eb47e6baeac1248" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35693 } } }
I20260619 16:52:30.035854 29935 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d089bd0bf2465e9eb47e6baeac1248 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:30.036659 29943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:30.046900 29943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:30.048503 29943 catalog_manager.cc:1259] Loaded cluster ID: ea3b4bc77c314db98220f19895c6e15d
I20260619 16:52:30.048623 29943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 16:52:30.050772 29943 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd841fad9ad google::LogMessage::Fail() at ??:0
@ 0x7fd841fb088a google::LogMessage::SendToLog() at ??:0
@ 0x7fd841fad4de google::LogMessage::Flush() at ??:0
@ 0x7fd841faea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd8579e0512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd8579dd976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd857a65610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd857a65439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd844bf9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd844bfc91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd844bfc799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd844bd7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd847eb16db start_thread at ??:0
@ 0x7fd840e4761f clone at ??:0
W20260619 16:52:30.353232 29931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:30.353674 29931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:30.353886 29931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:30.368075 29931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:30.368204 29931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:30.368321 29931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:52:30.378729 29931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:41061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=36035
--tserver_master_addrs=127.14.153.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:30.380198 29931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:30.381497 29931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:30.394860 29971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:30.394903 29972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:30.395268 29974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:30.396384 29931 server_base.cc:1034] running on GCE node
I20260619 16:52:30.397029 29931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:30.398090 29931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:30.399356 29931 hybrid_clock.cc:648] HybridClock initialized: now 1781887950399313 us; error 59 us; skew 500 ppm
I20260619 16:52:30.399712 29931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:30 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:30.402843 29931 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:52:30.404408 29931 webserver.cc:460] Webserver started at http://127.14.153.1:36035/ using document root <none> and password file <none>
I20260619 16:52:30.405117 29931 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:30.405267 29931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:30.410897 29931 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:52:30.413964 29981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:30.415552 29931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:52:30.415696 29931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "7807891d5621493daac221d4d4b44dff"
format_stamp: "Formatted at 2026-06-19 16:52:24 on dist-test-slave-qb36"
server_key: "6c2e7cbd183d82ea5c760c17142aead1"
server_key_iv: "1780752c64fed76f729bdbc6a825d889"
server_key_version: "encryptionkey@0"
I20260619 16:52:30.416728 29931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:30.434504 29931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:30.438831 29931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:30.439134 29931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:30.440081 29931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:30.441390 29931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:30.441488 29931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:30.441561 29931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:30.441618 29931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:30.475167 29931 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:41061
I20260619 16:52:30.475234 30093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:41061 every 8 connection(s)
I20260619 16:52:30.476871 29931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:52:30.480054 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 29931
I20260619 16:52:30.480890 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:41313
--local_ip_for_outbound_sockets=127.14.153.2
--tserver_master_addrs=127.14.153.62:35693
--webserver_port=42085
--webserver_interface=127.14.153.2
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:30.482255 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:30.480224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:51617)
0619 16:52:30.481090 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:30.481359 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":340,"threads_started":1}
W20260619 16:52:30.483134 30094 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:35693 (0 consecutive failures): Network error: Failed to ping master at 127.14.153.62:35693: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
I20260619 16:52:30.484037 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:30.483680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:59573)
0619 16:52:30.483843 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:30.483925 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:52:30.484890 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:30.484556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:48091)
0619 16:52:30.484696 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:30.484780 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:52:30.485097 30094 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:52:30.811235 30097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:30.811522 30097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:30.811691 30097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:30.819416 30097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:30.819577 30097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:30.819763 30097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:52:30.830217 30097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:41313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=42085
--tserver_master_addrs=127.14.153.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:30.831588 30097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:30.832974 30097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:30.846036 30102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:30.846021 30103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:30.845983 30105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:30.846997 30097 server_base.cc:1034] running on GCE node
I20260619 16:52:30.847638 30097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:30.848583 30097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:30.849803 30097 hybrid_clock.cc:648] HybridClock initialized: now 1781887950849740 us; error 62 us; skew 500 ppm
I20260619 16:52:30.850118 30097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:30 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:30.853341 30097 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:52:30.854862 30097 webserver.cc:460] Webserver started at http://127.14.153.2:42085/ using document root <none> and password file <none>
I20260619 16:52:30.855634 30097 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:30.855790 30097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:30.861317 30097 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:52:30.864117 30112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:30.865442 30097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:52:30.865619 30097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "ff45e17bc9234608832517a470d62569"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e409d0eb7363f5299af9691698c6683"
server_key_iv: "c315bdeda5a93f2a4039403d46e69fe5"
server_key_version: "encryptionkey@0"
I20260619 16:52:30.866622 30097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:30.897758 30097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:30.904328 30097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:30.904666 30097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:30.906054 30097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:30.908144 30097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:30.908275 30097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:30.908420 30097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:30.908514 30097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:30.943174 30097 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:41313
I20260619 16:52:30.943256 30224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:41313 every 8 connection(s)
I20260619 16:52:30.944655 30097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:52:30.949606 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:30.947770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:43643)
0619 16:52:30.948478 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:30.948729 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":267,"threads_started":1}
W20260619 16:52:30.950336 30225 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:35693 (0 consecutive failures): Network error: Failed to ping master at 127.14.153.62:35693: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
I20260619 16:52:30.951202 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:30.950844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:55435)
0619 16:52:30.950978 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:30.951093 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:52:30.952080 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:30.951704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:57045)
0619 16:52:30.951818 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:30.951924 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 16:52:30.952332 30225 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:52:30.954267 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 30097
I20260619 16:52:30.954888 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:32771
--local_ip_for_outbound_sockets=127.14.153.3
--tserver_master_addrs=127.14.153.62:35693
--webserver_port=46281
--webserver_interface=127.14.153.3
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:31.287177 30228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:31.287478 30228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:31.287617 30228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:31.295254 30228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:31.295372 30228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:31.295571 30228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:52:31.305511 30228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:32771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=46281
--tserver_master_addrs=127.14.153.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:31.306929 30228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:31.308317 30228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:31.321667 30236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:31.321722 30234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:31.321739 30233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:31.322474 30228 server_base.cc:1034] running on GCE node
I20260619 16:52:31.323158 30228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:31.324184 30228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:31.325420 30228 hybrid_clock.cc:648] HybridClock initialized: now 1781887951325361 us; error 56 us; skew 500 ppm
I20260619 16:52:31.325766 30228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:31 dist-test-slave-qb36 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:31.329054 30228 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:52:31.330462 30228 webserver.cc:460] Webserver started at http://127.14.153.3:46281/ using document root <none> and password file <none>
I20260619 16:52:31.331198 30228 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:31.331306 30228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:31.336581 30228 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:52:31.339313 30243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:31.340672 30228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:52:31.340860 30228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "7aaf50a8c6e342c2a29fa5a996e6f22b"
format_stamp: "Formatted at 2026-06-19 16:52:25 on dist-test-slave-qb36"
server_key: "3e315941de4d7a13424432d46fe9ed38"
server_key_iv: "784937e906c77f45dc7f4b050894ea56"
server_key_version: "encryptionkey@0"
I20260619 16:52:31.341836 30228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:31.355413 30228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:31.359701 30228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:31.359985 30228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:31.360924 30228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:31.362190 30228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:31.362287 30228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:31.362434 30228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:31.362514 30228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:31.395579 30228 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:32771
I20260619 16:52:31.395664 30355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:32771 every 8 connection(s)
I20260619 16:52:31.397045 30228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:31.398044 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 30228
I20260619 16:52:31.402101 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:31.400111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:40447)
0619 16:52:31.400808 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:31.401011 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":271,"threads_started":1}
W20260619 16:52:31.402806 30356 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:35693 (0 consecutive failures): Network error: Failed to ping master at 127.14.153.62:35693: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
I20260619 16:52:31.403641 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:31.403311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:49969)
0619 16:52:31.403458 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:31.403535 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:52:31.404444 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:31.404133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:49591)
0619 16:52:31.404275 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:31.404345 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 16:52:31.404670 30356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:52:31.486550 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:31.485670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:37827)
0619 16:52:31.486192 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:31.486315 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":241,"threads_started":1}
I20260619 16:52:31.953851 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:31.952979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:32845)
0619 16:52:31.953491 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:31.953593 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":266,"threads_started":1}
I20260619 16:52:32.406100 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:32.405284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:54287)
0619 16:52:32.405757 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:32.405875 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":222,"threads_started":1}
I20260619 16:52:32.488445 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:32.487676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:37763)
0619 16:52:32.488172 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:32.488270 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":232,"threads_started":1}
I20260619 16:52:32.955881 30364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:32.954944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:37381)
0619 16:52:32.955607 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:32.955699 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":413,"threads_started":1}
I20260619 16:52:33.407886 30365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:33.407186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:42039)
0619 16:52:33.407635 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:33.407720 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":219,"threads_started":1}
I20260619 16:52:33.490337 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:33.489542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:40657)
0619 16:52:33.490044 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:33.490134 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
I20260619 16:52:33.957690 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:33.956857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:40577)
0619 16:52:33.957408 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:33.957493 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":318,"threads_started":1}
I20260619 16:52:34.409471 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:34.408775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:38601)
0619 16:52:34.409209 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:34.409289 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260619 16:52:34.492236 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:34.491486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:44867)
0619 16:52:34.491948 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:34.492033 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":237,"threads_started":1}
I20260619 16:52:34.959580 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:34.958693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:43119)
0619 16:52:34.959313 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:34.959395 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":253,"threads_started":1}
I20260619 16:52:35.411437 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:35.410491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:53763)
0619 16:52:35.411128 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:35.411239 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":289,"threads_started":1}
I20260619 16:52:35.494030 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:35.493290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:55587)
0619 16:52:35.493774 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:35.493860 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
I20260619 16:52:35.961421 30373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:35.960640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:32905)
0619 16:52:35.961147 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:35.961244 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":250,"threads_started":1}
I20260619 16:52:36.413431 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:36.412520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:46967)
0619 16:52:36.413117 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:36.413252 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":321,"threads_started":1}
I20260619 16:52:36.495761 30375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:36.494992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:42279)
0619 16:52:36.495492 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:36.495585 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":224,"threads_started":1}
I20260619 16:52:36.963464 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:36.962458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:51433)
0619 16:52:36.963089 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:36.963223 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":332,"threads_started":1}
I20260619 16:52:37.415127 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:37.414293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:56363)
0619 16:52:37.414780 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:37.414871 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260619 16:52:37.497730 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:37.496907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:38459)
0619 16:52:37.497416 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:37.497504 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":238,"threads_started":1}
I20260619 16:52:37.965214 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:37.964402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:37443)
0619 16:52:37.964904 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:37.964987 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":215,"threads_started":1}
I20260619 16:52:38.417083 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.416243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:58233)
0619 16:52:38.416753 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.416855 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":235,"threads_started":1}
I20260619 16:52:38.499773 30381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.498811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:52355)
0619 16:52:38.499493 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.499608 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":398,"threads_started":1}
I20260619 16:52:38.966960 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.966226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:39697)
0619 16:52:38.966700 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.966797 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260619 16:52:39.418879 30383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:39.418152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:58989)
0619 16:52:39.418613 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:39.418716 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":245,"threads_started":1}
I20260619 16:52:39.501498 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:39.500756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:46641)
0619 16:52:39.501237 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:39.501323 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":235,"threads_started":1}
I20260619 16:52:39.968730 30385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:39.967905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:47405)
0619 16:52:39.968392 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:39.968475 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260619 16:52:40.420725 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:40.419938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:36341)
0619 16:52:40.420455 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:40.420543 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":309,"threads_started":1}
I20260619 16:52:40.503484 30387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:40.502566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:58697)
0619 16:52:40.503210 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:40.503308 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":249,"threads_started":1}
I20260619 16:52:40.970851 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:40.969927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:59111)
0619 16:52:40.970471 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:40.970562 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":325,"threads_started":1}
I20260619 16:52:41.422506 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:41.421783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:38993)
0619 16:52:41.422261 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:41.422346 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":268,"threads_started":1}
I20260619 16:52:41.505071 30390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:41.504374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:55329)
0619 16:52:41.504807 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:41.504895 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":224,"threads_started":1}
I20260619 16:52:41.972733 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:41.972000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:35679)
0619 16:52:41.972447 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:41.972547 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":216,"threads_started":1}
I20260619 16:52:42.424220 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:42.423501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:45271)
0619 16:52:42.423939 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:42.424028 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":209,"threads_started":1}
I20260619 16:52:42.506736 30393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:42.506074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:51883)
0619 16:52:42.506502 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:42.506582 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":222,"threads_started":1}
I20260619 16:52:42.974422 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:42.973681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:33115)
0619 16:52:42.974171 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:42.974261 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":237,"threads_started":1}
I20260619 16:52:43.425928 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:43.425224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:42539)
0619 16:52:43.425688 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:43.425775 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":242,"threads_started":1}
I20260619 16:52:43.508435 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:43.507696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:40245)
0619 16:52:43.508166 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:43.508253 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":238,"threads_started":1}
I20260619 16:52:43.976248 30397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:43.975433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:50589)
0619 16:52:43.975929 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:43.976046 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":238,"threads_started":1}
I20260619 16:52:44.427884 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:44.426954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:52463)
0619 16:52:44.427594 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:44.427705 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":373,"threads_started":1}
I20260619 16:52:44.510461 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:44.509564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:40239)
0619 16:52:44.510133 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:44.510226 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":304,"threads_started":1}
I20260619 16:52:44.977972 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:44.977226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:43247)
0619 16:52:44.977687 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:44.977776 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":225,"threads_started":1}
I20260619 16:52:45.429600 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:45.428879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:45031)
0619 16:52:45.429340 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:45.429424 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":231,"threads_started":1}
I20260619 16:52:45.512543 30402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:45.511738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.1:53929)
0619 16:52:45.512274 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:45.512361 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":306,"threads_started":1}
I20260619 16:52:45.979956 30403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:45.979228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.2:59437)
0619 16:52:45.979665 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:45.979776 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260619 16:52:46.401784 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 29931
I20260619 16:52:46.424054 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:46.403212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.14.153.1:41061)
0619 16:52:46.403816 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:46.403838 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:46.403886 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:52:46.405673 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:46.405684 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:46.406151 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:46.406451 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:46.407933 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:46.409214 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:46.410529 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:46.411420 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:46.411569 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:46.417161 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:46.417172 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:46.417191 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:46.417242 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:46.420465 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:46.421357 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:46.421372 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:46.421379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:46.421555 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:46.421917 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:46.421920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:46.421922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:46.422470 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:46.422819 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:46.423331 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 16:52:46.423772 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":243,"threads_started":1}
I20260619 16:52:46.425705 30089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:46.431542 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:46.430725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35693 (local address 127.14.153.3:46737)
0619 16:52:46.431272 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:46.431380 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":323,"threads_started":1}
I20260619 16:52:46.510358 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 29931
I20260619 16:52:46.528178 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 30097
I20260619 16:52:46.546411 30407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:46.529308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.14.153.2:41313)
0619 16:52:46.529805 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:46.529820 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:46.529855 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:52:46.531512 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:46.531523 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:46.531971 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:46.532329 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:46.533852 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:46.535181 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:46.536480 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:46.537182 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:46.537381 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:46.540150 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:46.540158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:46.540174 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:46.540213 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:46.542950 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:46.543789 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:46.543796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:46.543803 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:46.543982 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:46.544414 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:46.544418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:46.544420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:46.545034 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:46.545313 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:46.545634 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:52:46.546082 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
I20260619 16:52:46.548038 30220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:46.631189 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 30097
I20260619 16:52:46.649528 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 30228
I20260619 16:52:46.667944 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:46.650629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.14.153.3:32771)
0619 16:52:46.651318 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:46.651333 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:46.651375 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:52:46.653077 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:46.653088 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:46.653545 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:46.653857 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:46.655382 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:46.656780 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:46.658144 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:46.658814 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:46.658956 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:46.661680 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:46.661688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:46.661704 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:46.661742 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:46.664789 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:46.665527 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:46.665535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:46.665541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:46.665718 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:46.666063 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:46.666067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:46.666069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:46.666595 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:46.666840 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:46.667218 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 16:52:46.667564 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":303,"threads_started":1}
I20260619 16:52:46.669319 30351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:46.751405 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 30228
2026-06-19T16:52:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24433 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:52:46 dist-test-slave-qb36 krb5kdc[30413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:52:46 dist-test-slave-qb36 krb5kdc[30413](info): set up 2 sockets
Jun 19 16:52:46 dist-test-slave-qb36 krb5kdc[30413](info): commencing operation
krb5kdc: starting...
W20260619 16:52:47.855801 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:52:48 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:52:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:52:48.101017 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42119
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:42119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:48.435726 30426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:48.436100 30426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:48.436282 30426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:48.443636 30426 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:52:48.443733 30426 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:52:48.443801 30426 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:52:48.443878 30426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:48.443969 30426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:48.444096 30426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:48.444149 30426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:48.444216 30426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:48.454233 30426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:42119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:48.455689 30426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:48.457233 30426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:48.469614 30431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:48.469578 30432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:48.469527 30434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:48.469750 30426 server_base.cc:1034] running on GCE node
I20260619 16:52:48.471019 30426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:48.472536 30426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:48.473778 30426 hybrid_clock.cc:648] HybridClock initialized: now 1781887968473699 us; error 59 us; skew 500 ppm
I20260619 16:52:48.474136 30426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:48 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:48.477347 30426 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:52:48.478752 30426 webserver.cc:460] Webserver started at http://127.14.153.62:38201/ using document root <none> and password file <none>
I20260619 16:52:48.479525 30426 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:48.479658 30426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:48.479972 30426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:48.483008 30426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "0cf2eb17de13433fa0f96f14553c6092"
format_stamp: "Formatted at 2026-06-19 16:52:48 on dist-test-slave-qb36"
server_key: "44da3b9d2daf5d1628faa298f76d0b49"
server_key_iv: "43bc98210ea9984c41cdac3073cfbb92"
server_key_version: "encryptionkey@0"
I20260619 16:52:48.483960 30426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "0cf2eb17de13433fa0f96f14553c6092"
format_stamp: "Formatted at 2026-06-19 16:52:48 on dist-test-slave-qb36"
server_key: "44da3b9d2daf5d1628faa298f76d0b49"
server_key_iv: "43bc98210ea9984c41cdac3073cfbb92"
server_key_version: "encryptionkey@0"
I20260619 16:52:48.489332 30426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 16:52:48.492885 30441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:48.494354 30426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:52:48.494491 30426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "0cf2eb17de13433fa0f96f14553c6092"
format_stamp: "Formatted at 2026-06-19 16:52:48 on dist-test-slave-qb36"
server_key: "44da3b9d2daf5d1628faa298f76d0b49"
server_key_iv: "43bc98210ea9984c41cdac3073cfbb92"
server_key_version: "encryptionkey@0"
I20260619 16:52:48.494695 30426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:48.523425 30426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:48.527555 30426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:48.527786 30426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:48.546509 30426 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:42119
I20260619 16:52:48.546533 30492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:42119 every 8 connection(s)
I20260619 16:52:48.547979 30426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:48.551976 30493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:48.554850 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 30426
I20260619 16:52:48.555133 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:52:48.555758 14948 external_mini_cluster.cc:1428] Setting key 6ef011b70785773c02d088b2dd472163
I20260619 16:52:48.565522 30493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Bootstrap starting.
I20260619 16:52:48.569341 30493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:48.570401 30493 log.cc:824] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:52:48 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:48.573514 30493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: No bootstrap required, opened a new log
I20260619 16:52:48.580869 30496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:48.557950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.14.153.62:42119)
0619 16:52:48.558761 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:48.558780 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:48.558835 (+ 55us) server_negotiation.cc:407] Connection header received
0619 16:52:48.561143 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:48.561182 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:48.561763 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:48.562269 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:48.563828 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:48.565809 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:48.567333 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:48.568402 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:48.568614 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:48.572979 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:48.572988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:48.573004 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:48.573047 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:48.576325 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:48.577166 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:48.577174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:48.577181 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:48.577365 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:48.577801 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:48.577805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:48.577807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:48.578439 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:48.578806 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:48.579327 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 16:52:48.579772 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":309,"threads_started":1}
I20260619 16:52:48.582062 30493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:52:48.582585 30493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:48.582710 30493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf2eb17de13433fa0f96f14553c6092, State: Initialized, Role: FOLLOWER
I20260619 16:52:48.583377 30493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:52:48.583642 30493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:48.583797 30493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:48.583992 30493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:48.586310 30493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:52:48.586786 30493 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf2eb17de13433fa0f96f14553c6092; no voters:
I20260619 16:52:48.587473 30493 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:52:48.587620 30498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:48.588446 30498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf2eb17de13433fa0f96f14553c6092, State: Running, Role: LEADER
I20260619 16:52:48.588982 30498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:52:48.589756 30493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:48.591888 30500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf2eb17de13433fa0f96f14553c6092. Latest consensus state: current_term: 1 leader_uuid: "0cf2eb17de13433fa0f96f14553c6092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } } }
I20260619 16:52:48.592391 30500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:48.593171 30499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cf2eb17de13433fa0f96f14553c6092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } } }
I20260619 16:52:48.598233 30499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:48.600966 30506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:48.606457 30506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:48.607685 30515 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:52:48.615630 30506 catalog_manager.cc:1347] Generated new cluster ID: b257063516d54a04a160bfafce01bbe9
I20260619 16:52:48.615767 30506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:48.635000 30506 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:52:48.636464 30506 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:48.647011 30506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Generated new TSK 0
I20260619 16:52:48.647938 30506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:52:48.707779 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42119
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:49.045917 30519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:49.046211 30519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:49.046382 30519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:49.053874 30519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:49.054067 30519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:49.054257 30519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:52:49.064649 30519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:49.066077 30519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:49.067529 30519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:49.081499 30524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:49.081547 30525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:49.081914 30527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:49.082628 30519 server_base.cc:1034] running on GCE node
I20260619 16:52:49.083384 30519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:49.084385 30519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:49.085593 30519 hybrid_clock.cc:648] HybridClock initialized: now 1781887969085541 us; error 46 us; skew 500 ppm
I20260619 16:52:49.085961 30519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:49 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887969, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:49.089332 30519 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:52:49.091156 30519 webserver.cc:460] Webserver started at http://127.14.153.1:40789/ using document root <none> and password file <none>
I20260619 16:52:49.091931 30519 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:49.092087 30519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:49.092401 30519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:49.095570 30519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "17881fb24f6f42a7a2092fd7266ca89e"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "af2e8f45e02bd439fb2184c81bb5ec03"
server_key_iv: "04a2c7484aced51d94954466ccfb264a"
server_key_version: "encryptionkey@0"
I20260619 16:52:49.096405 30519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "17881fb24f6f42a7a2092fd7266ca89e"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "af2e8f45e02bd439fb2184c81bb5ec03"
server_key_iv: "04a2c7484aced51d94954466ccfb264a"
server_key_version: "encryptionkey@0"
I20260619 16:52:49.102167 30519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:52:49.106673 30534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:49.108141 30519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:52:49.108322 30519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "17881fb24f6f42a7a2092fd7266ca89e"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "af2e8f45e02bd439fb2184c81bb5ec03"
server_key_iv: "04a2c7484aced51d94954466ccfb264a"
server_key_version: "encryptionkey@0"
I20260619 16:52:49.108530 30519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:49.133176 30519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:49.137634 30519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:49.137970 30519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:49.139093 30519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:49.140540 30519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:49.140650 30519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:49.140806 30519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:49.140882 30519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:49.174804 30519 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:39035
I20260619 16:52:49.174862 30646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:39035 every 8 connection(s)
I20260619 16:52:49.176234 30519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:52:49.184966 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 30519
I20260619 16:52:49.185206 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:52:49.185871 14948 external_mini_cluster.cc:1428] Setting key 8504a56fca01fe13d10baee2319fc629
Jun 19 16:52:49 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887969, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:49.197883 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:49.178914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:42307 (local address 127.14.153.62:42119)
0619 16:52:49.179393 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:49.179398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:49.180307 (+ 909us) server_negotiation.cc:407] Connection header received
0619 16:52:49.182179 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:49.182185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:49.182281 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:49.182402 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:49.184087 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:49.184832 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:49.186655 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:49.187166 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:49.187311 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:49.191787 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:49.191792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:49.191795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:49.191830 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:49.194458 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:49.195452 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:49.195456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:49.195459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:49.195593 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:49.196057 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:49.196063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:49.196066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:49.196344 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:49.196507 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:49.197245 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 16:52:49.197481 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":371,"thread_start_us":270,"threads_started":1}
I20260619 16:52:49.198920 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:49.179396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:42307)
0619 16:52:49.180132 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:49.180176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:49.181885 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:49.182665 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:49.182674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:49.183293 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:49.183895 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:49.183913 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:49.184988 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:49.184998 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:49.186457 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:49.186473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:49.187399 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:49.187407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:49.187733 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:49.188537 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:49.188588 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:49.191548 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:49.194623 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:49.194636 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:49.194659 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:49.195298 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:49.195710 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:49.195715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:49.195718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:49.195924 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:49.196483 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:49.196491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:49.196851 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 16:52:49.197131 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 16:52:49.197700 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":289,"threads_started":1}
I20260619 16:52:49.200603 30647 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42119
I20260619 16:52:49.200940 30647 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:49.201886 30647 heartbeater.cc:511] Master 127.14.153.62:42119 requested a full tablet report, sending...
I20260619 16:52:49.204170 30458 ts_manager.cc:194] Registered new tserver with Master: 17881fb24f6f42a7a2092fd7266ca89e (127.14.153.1:39035)
I20260619 16:52:49.205670 30458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:42307
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:52:49.242813 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42119
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:49.583154 30653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:49.583549 30653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:49.583724 30653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:49.591291 30653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:49.591456 30653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:49.591642 30653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:52:49.601704 30653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:49.603159 30653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:49.604601 30653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:49.618142 30658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:49.618245 30659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:49.618271 30661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:49.619218 30653 server_base.cc:1034] running on GCE node
I20260619 16:52:49.619958 30653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:49.620975 30653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:49.622186 30653 hybrid_clock.cc:648] HybridClock initialized: now 1781887969622117 us; error 63 us; skew 500 ppm
I20260619 16:52:49.622565 30653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:49 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887969, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:49.625757 30653 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:52:49.627350 30653 webserver.cc:460] Webserver started at http://127.14.153.2:36383/ using document root <none> and password file <none>
I20260619 16:52:49.628060 30653 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:49.628209 30653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:49.628580 30653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:49.631583 30653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "a9ea7335538d4aa9aa5409162f143ad0"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "c1e6e7fb71cddb5cb70ad9b852f862f1"
server_key_iv: "8c3a5cdb6ca33d9a477e7a6f884e51e6"
server_key_version: "encryptionkey@0"
I20260619 16:52:49.632388 30653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "a9ea7335538d4aa9aa5409162f143ad0"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "c1e6e7fb71cddb5cb70ad9b852f862f1"
server_key_iv: "8c3a5cdb6ca33d9a477e7a6f884e51e6"
server_key_version: "encryptionkey@0"
I20260619 16:52:49.637745 30653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:52:49.641301 30668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:49.642604 30653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:52:49.642781 30653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "a9ea7335538d4aa9aa5409162f143ad0"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "c1e6e7fb71cddb5cb70ad9b852f862f1"
server_key_iv: "8c3a5cdb6ca33d9a477e7a6f884e51e6"
server_key_version: "encryptionkey@0"
I20260619 16:52:49.642977 30653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:49.660094 30653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:49.664366 30653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:49.664669 30653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:49.665748 30653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:49.667066 30653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:49.667171 30653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:49.667321 30653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:49.667408 30653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:49.701424 30653 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:42807
I20260619 16:52:49.701481 30780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:42807 every 8 connection(s)
I20260619 16:52:49.702782 30653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:52:49.706758 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 30653
I20260619 16:52:49.706969 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:52:49.707464 14948 external_mini_cluster.cc:1428] Setting key ebcccdd15be7f1769d20f39278d248db
Jun 19 16:52:49 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887969, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:49.723588 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:49.705346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:48605 (local address 127.14.153.62:42119)
0619 16:52:49.705788 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:49.705794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:49.706824 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 16:52:49.708893 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:49.708899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:49.708985 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:49.709112 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:49.710672 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:49.711489 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:49.713141 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:49.713570 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:49.713701 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:49.717767 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:49.717773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:49.717776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:49.717809 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:49.720482 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:49.721220 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:49.721227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:49.721231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:49.721362 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:49.721789 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:49.721795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:49.721798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:49.722072 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:49.722224 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:49.722942 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 16:52:49.723206 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":206,"threads_started":1}
I20260619 16:52:49.724581 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:49.705899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:48605)
0619 16:52:49.706629 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:49.706674 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:49.708628 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:49.709350 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:49.709366 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:49.709912 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:49.710509 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:49.710526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:49.711651 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:49.711656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:49.712990 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:49.713001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:49.713795 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:49.713802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:49.714038 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:49.714807 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:49.714843 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:49.717518 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:49.720632 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:49.720641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:49.720659 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:49.721101 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:49.721482 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:49.721488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:49.721491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:49.721679 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:49.722196 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:49.722204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:49.722552 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 16:52:49.722857 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 16:52:49.723451 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":330,"threads_started":1}
I20260619 16:52:49.726153 30781 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42119
I20260619 16:52:49.726511 30781 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:49.727589 30781 heartbeater.cc:511] Master 127.14.153.62:42119 requested a full tablet report, sending...
I20260619 16:52:49.729454 30458 ts_manager.cc:194] Registered new tserver with Master: a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807)
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
I20260619 16:52:49.730747 30458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:48605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:52:49.762542 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42119
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:50.094313 30787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:50.094827 30787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:50.095017 30787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:50.103597 30787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:50.103718 30787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:50.103888 30787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:52:50.114006 30787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:52:50.115643 30787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:50.117039 30787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:50.130662 30792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:50.130690 30793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:50.131382 30795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:50.132314 30787 server_base.cc:1034] running on GCE node
I20260619 16:52:50.132992 30787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:50.134016 30787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:50.135234 30787 hybrid_clock.cc:648] HybridClock initialized: now 1781887970135169 us; error 61 us; skew 500 ppm
I20260619 16:52:50.135569 30787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:52:50 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:50.138871 30787 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:52:50.140443 30787 webserver.cc:460] Webserver started at http://127.14.153.3:41971/ using document root <none> and password file <none>
I20260619 16:52:50.141182 30787 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:50.141332 30787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:50.141649 30787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:50.144820 30787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "a3fe0621da7545ff89b52ff080526ea9"
format_stamp: "Formatted at 2026-06-19 16:52:50 on dist-test-slave-qb36"
server_key: "401f7d0c0c8076949cecadebda8267f3"
server_key_iv: "7f7c472c5aa054bce3ee1d9a8fa87eed"
server_key_version: "encryptionkey@0"
I20260619 16:52:50.145622 30787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "a3fe0621da7545ff89b52ff080526ea9"
format_stamp: "Formatted at 2026-06-19 16:52:50 on dist-test-slave-qb36"
server_key: "401f7d0c0c8076949cecadebda8267f3"
server_key_iv: "7f7c472c5aa054bce3ee1d9a8fa87eed"
server_key_version: "encryptionkey@0"
I20260619 16:52:50.151198 30787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:52:50.154793 30802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:50.156247 30787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:52:50.156433 30787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "a3fe0621da7545ff89b52ff080526ea9"
format_stamp: "Formatted at 2026-06-19 16:52:50 on dist-test-slave-qb36"
server_key: "401f7d0c0c8076949cecadebda8267f3"
server_key_iv: "7f7c472c5aa054bce3ee1d9a8fa87eed"
server_key_version: "encryptionkey@0"
I20260619 16:52:50.156659 30787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:50.187754 30787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:50.192309 30787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:50.192617 30787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:50.193619 30787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:50.194970 30787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:50.195094 30787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 16:52:50.195246 30787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:50.195328 30787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:50.209703 30647 heartbeater.cc:503] Master 127.14.153.62:42119 was elected leader, sending a full tablet report...
I20260619 16:52:50.229672 30787 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:42727
I20260619 16:52:50.229729 30914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:42727 every 8 connection(s)
I20260619 16:52:50.231015 30787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:50.236342 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 30787
I20260619 16:52:50.236558 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:52:50.236994 14948 external_mini_cluster.cc:1428] Setting key 6a35572626aa5cbeb6c687c1f0a84dd9
Jun 19 16:52:50 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:52:50.251308 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.233792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:32893 (local address 127.14.153.62:42119)
0619 16:52:50.234273 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.234277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.235085 (+ 808us) server_negotiation.cc:407] Connection header received
0619 16:52:50.237104 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.237110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.237232 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.237362 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:50.238894 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.239838 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.241396 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.241783 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.241913 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.245381 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:50.245386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:50.245389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:50.245422 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:50.248307 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.249002 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.249006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.249008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.249126 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.249493 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.249496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.249498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.249763 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:50.249876 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.250654 (+ 778us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.250885 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":248,"threads_started":1}
I20260619 16:52:50.252501 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.234202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:32893)
0619 16:52:50.234884 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:50.234925 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:50.236812 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:50.237624 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:50.237633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:50.238153 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:50.238720 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.238738 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.240004 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.240008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.241249 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.241261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.242014 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.242021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.242261 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.242954 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:50.242978 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:50.245167 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:50.248454 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:50.248462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:50.248475 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:50.248898 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:50.249219 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:50.249222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:50.249224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:50.249404 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:50.249874 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:50.249880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:50.250238 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 16:52:50.250520 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 16:52:50.251103 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":281,"threads_started":1}
I20260619 16:52:50.253823 30915 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42119
I20260619 16:52:50.254142 30915 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:50.254843 30915 heartbeater.cc:511] Master 127.14.153.62:42119 requested a full tablet report, sending...
I20260619 16:52:50.256255 30458 ts_manager.cc:194] Registered new tserver with Master: a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727)
I20260619 16:52:50.257016 30458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:32893
I20260619 16:52:50.264746 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:52:50.281657 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.269272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.14.153.62:42119)
0619 16:52:50.269464 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.269468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.269779 (+ 311us) server_negotiation.cc:407] Connection header received
0619 16:52:50.269910 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.269914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.269979 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.270069 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:50.271497 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.272200 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.273558 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.273968 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.274073 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.276527 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:50.276532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:50.276535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:50.276566 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:50.279209 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.279832 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.279837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.279839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.280035 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.280387 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.280390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.280392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.280706 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:50.280974 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.281233 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.281407 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:52:50.285925 30458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:52:50.293134 30458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:52:50.320854 30928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.305881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:42807 (local address 127.0.0.1:36770)
0619 16:52:50.306705 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:50.306750 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:50.306944 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:50.308962 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:50.308974 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:50.309017 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:50.309529 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.309547 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.312513 (+ 2966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.312522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.316085 (+ 3563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.316118 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.319340 (+ 3222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.319350 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.319686 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.319767 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 16:52:50.319959 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:52:50.320266 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":313,"threads_started":1}
I20260619 16:52:50.322292 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.306003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.14.153.2:42807)
0619 16:52:50.307796 (+ 1793us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.307810 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.307847 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:52:50.307984 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.307997 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.308350 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.308606 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:50.310174 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.312300 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.316422 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.318633 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.318831 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.319157 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.321559 (+ 2402us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.321841 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":886,"threads_started":1}
I20260619 16:52:50.331723 30928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.307737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:42727 (local address 127.0.0.1:33682)
0619 16:52:50.321175 (+ 13438us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:50.321194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:50.321328 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:50.324411 (+ 3083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:50.324419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:50.324459 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:50.324758 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.324773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.327608 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.327613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.329541 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.329554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.330907 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.330913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.331141 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.331180 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:52:50.331249 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:52:50.331405 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13307,"thread_start_us":9314,"threads_started":1}
I20260619 16:52:50.331394 30716 tablet_service.cc:1467] Processing CreateTablet for tablet cc0d119a696c427e83e5b480907118cd (DEFAULT_TABLE table=test-table [id=289a3680b26d4f9caf6c1251fa75def7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:50.332463 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.307278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.0.0.1:38872)
0619 16:52:50.317410 (+ 10132us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:50.317433 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:50.317572 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:50.324472 (+ 6900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:50.324477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:50.324534 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:50.324956 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.324967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.327017 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.327048 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.329437 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.329458 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.331608 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.331615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.331797 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.331843 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 16:52:50.331938 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:52:50.332070 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9976,"spinlock_wait_cycles":4480,"thread_start_us":19960,"threads_started":1}
I20260619 16:52:50.332791 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.308514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.14.153.3:42727)
0619 16:52:50.323563 (+ 15049us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.323576 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.323620 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:52:50.323763 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.323775 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.324120 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.325000 (+ 880us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:50.325363 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.327448 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.329701 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.330713 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.332022 (+ 1309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.332224 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.332285 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.332437 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14807,"thread_start_us":14587,"threads_started":1}
I20260619 16:52:50.333173 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.307528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.14.153.1:39035)
0619 16:52:50.323539 (+ 16011us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.323547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.323574 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:52:50.323660 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.323666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.323970 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.325144 (+ 1174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:50.325313 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.326884 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.329634 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.331057 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.331198 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.331443 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.332691 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.332860 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15831,"thread_start_us":16611,"threads_started":1}
I20260619 16:52:50.333365 30716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0d119a696c427e83e5b480907118cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:50.336975 30582 tablet_service.cc:1467] Processing CreateTablet for tablet cc0d119a696c427e83e5b480907118cd (DEFAULT_TABLE table=test-table [id=289a3680b26d4f9caf6c1251fa75def7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:50.336975 30850 tablet_service.cc:1467] Processing CreateTablet for tablet cc0d119a696c427e83e5b480907118cd (DEFAULT_TABLE table=test-table [id=289a3680b26d4f9caf6c1251fa75def7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:50.338541 30582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0d119a696c427e83e5b480907118cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:50.338547 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc0d119a696c427e83e5b480907118cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:50.346336 30934 tablet_bootstrap.cc:492] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: Bootstrap starting.
I20260619 16:52:50.350489 30934 tablet_bootstrap.cc:654] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:50.351720 30934 log.cc:824] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:50.353858 30936 tablet_bootstrap.cc:492] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: Bootstrap starting.
I20260619 16:52:50.354705 30937 tablet_bootstrap.cc:492] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: Bootstrap starting.
I20260619 16:52:50.355146 30934 tablet_bootstrap.cc:492] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: No bootstrap required, opened a new log
I20260619 16:52:50.355441 30934 ts_tablet_manager.cc:1397] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 16:52:50.358963 30937 tablet_bootstrap.cc:654] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:50.360366 30937 log.cc:824] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:50.362502 30936 tablet_bootstrap.cc:654] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:50.363792 30936 log.cc:824] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:50.364339 30937 tablet_bootstrap.cc:492] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: No bootstrap required, opened a new log
I20260619 16:52:50.364638 30937 ts_tablet_manager.cc:1397] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260619 16:52:50.366303 30934 raft_consensus.cc:348] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:52:50.366780 30934 raft_consensus.cc:374] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:50.366886 30934 raft_consensus.cc:729] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ea7335538d4aa9aa5409162f143ad0, State: Initialized, Role: FOLLOWER
I20260619 16:52:50.367125 30936 tablet_bootstrap.cc:492] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: No bootstrap required, opened a new log
I20260619 16:52:50.367466 30936 ts_tablet_manager.cc:1397] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260619 16:52:50.367694 30934 consensus_queue.cc:260] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:52:50.368897 30781 heartbeater.cc:503] Master 127.14.153.62:42119 was elected leader, sending a full tablet report...
I20260619 16:52:50.369547 30934 ts_tablet_manager.cc:1428] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 16:52:50.374662 30937 raft_consensus.cc:348] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:52:50.375172 30937 raft_consensus.cc:374] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:50.375284 30937 raft_consensus.cc:729] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17881fb24f6f42a7a2092fd7266ca89e, State: Initialized, Role: FOLLOWER
I20260619 16:52:50.376055 30937 consensus_queue.cc:260] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:52:50.376591 30936 raft_consensus.cc:348] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:52:50.376952 30936 raft_consensus.cc:374] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:50.377046 30936 raft_consensus.cc:729] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fe0621da7545ff89b52ff080526ea9, State: Initialized, Role: FOLLOWER
I20260619 16:52:50.377720 30936 consensus_queue.cc:260] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:52:50.378023 30937 ts_tablet_manager.cc:1428] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260619 16:52:50.378952 30915 heartbeater.cc:503] Master 127.14.153.62:42119 was elected leader, sending a full tablet report...
I20260619 16:52:50.379575 30936 ts_tablet_manager.cc:1428] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20260619 16:52:50.416946 30940 raft_consensus.cc:482] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:50.417389 30940 raft_consensus.cc:504] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:52:50.419185 30940 leader_election.cc:288] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17881fb24f6f42a7a2092fd7266ca89e (127.14.153.1:39035), a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727)
I20260619 16:52:50.426448 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.419595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:50721 (local address 127.14.153.1:39035)
0619 16:52:50.419801 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.419808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.420825 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 16:52:50.421054 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.421066 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.421156 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.421269 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:50.421859 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.422613 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.424673 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.425476 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.425672 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.425750 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.426052 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.426185 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:52:50.426441 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.419671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:58083 (local address 127.14.153.3:42727)
0619 16:52:50.419880 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.419888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.420917 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 16:52:50.421142 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.421146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.421207 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.421332 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:50.421959 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.422678 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.424637 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.425413 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.425603 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.425692 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.426042 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.426178 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:52:50.426652 30944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.419845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:42727 (local address 127.14.153.2:58083)
0619 16:52:50.420803 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:50.420818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:50.420974 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:50.421345 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:50.421350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:50.421377 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:50.421810 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.421818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.422821 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.422826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.424487 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.424499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.425675 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.425681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.425850 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.425901 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 16:52:50.426006 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:52:50.426142 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":33,"thread_start_us":520,"threads_started":1}
I20260619 16:52:50.427012 30943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.419792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.14.153.2:50721)
0619 16:52:50.420699 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:50.420719 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:50.420905 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:50.421283 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:50.421288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:50.421313 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:50.421701 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.421714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.422749 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.422756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.424509 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:50.424529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.425704 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.425710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:50.425895 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.425918 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:52:50.426022 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:52:50.426153 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":21,"thread_start_us":537,"threads_started":1}
I20260619 16:52:50.427845 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d119a696c427e83e5b480907118cd" candidate_uuid: "a9ea7335538d4aa9aa5409162f143ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fe0621da7545ff89b52ff080526ea9" is_pre_election: true
I20260619 16:52:50.428011 30602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d119a696c427e83e5b480907118cd" candidate_uuid: "a9ea7335538d4aa9aa5409162f143ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17881fb24f6f42a7a2092fd7266ca89e" is_pre_election: true
I20260619 16:52:50.428368 30870 raft_consensus.cc:2454] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ea7335538d4aa9aa5409162f143ad0 in term 0.
W20260619 16:52:50.428654 30648 tablet.cc:2369] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:50.428670 30602 raft_consensus.cc:2454] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ea7335538d4aa9aa5409162f143ad0 in term 0.
I20260619 16:52:50.429301 30670 leader_election.cc:302] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3fe0621da7545ff89b52ff080526ea9, a9ea7335538d4aa9aa5409162f143ad0; no voters:
I20260619 16:52:50.429694 30940 raft_consensus.cc:2781] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:52:50.429844 30940 raft_consensus.cc:482] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:52:50.429949 30940 raft_consensus.cc:3037] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:50.432293 30940 raft_consensus.cc:504] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:52:50.432953 30940 leader_election.cc:288] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [CANDIDATE]: Term 1 election: Requested vote from peers 17881fb24f6f42a7a2092fd7266ca89e (127.14.153.1:39035), a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727)
I20260619 16:52:50.433388 30602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d119a696c427e83e5b480907118cd" candidate_uuid: "a9ea7335538d4aa9aa5409162f143ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17881fb24f6f42a7a2092fd7266ca89e"
I20260619 16:52:50.433475 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc0d119a696c427e83e5b480907118cd" candidate_uuid: "a9ea7335538d4aa9aa5409162f143ad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3fe0621da7545ff89b52ff080526ea9"
I20260619 16:52:50.433636 30602 raft_consensus.cc:3037] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:50.433722 30870 raft_consensus.cc:3037] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:50.436061 30602 raft_consensus.cc:2454] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ea7335538d4aa9aa5409162f143ad0 in term 1.
I20260619 16:52:50.436115 30870 raft_consensus.cc:2454] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ea7335538d4aa9aa5409162f143ad0 in term 1.
I20260619 16:52:50.436573 30672 leader_election.cc:302] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17881fb24f6f42a7a2092fd7266ca89e, a9ea7335538d4aa9aa5409162f143ad0; no voters:
I20260619 16:52:50.436960 30940 raft_consensus.cc:2781] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:50.437412 30940 raft_consensus.cc:686] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 1 LEADER]: Becoming Leader. State: Replica: a9ea7335538d4aa9aa5409162f143ad0, State: Running, Role: LEADER
I20260619 16:52:50.437928 30940 consensus_queue.cc:237] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:52:50.442847 30457 catalog_manager.cc:5495] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2). New cstate: current_term: 1 leader_uuid: "a9ea7335538d4aa9aa5409162f143ad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } health_report { overall_health: UNKNOWN } } }
W20260619 16:52:50.454927 30782 tablet.cc:2369] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:52:50.482721 30916 tablet.cc:2369] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:50.485039 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.479146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.14.153.2:42807)
0619 16:52:50.479407 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.479412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.479432 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:52:50.479599 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.479603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.479684 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.479842 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:52:50.480282 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.481034 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.482603 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.483211 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.483328 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.483944 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:52:50.484484 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:52:50.484607 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.484713 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.484789 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 16:52:50.493520 30870 raft_consensus.cc:1261] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [term 1 FOLLOWER]: Refusing update from remote peer a9ea7335538d4aa9aa5409162f143ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:50.493533 30602 raft_consensus.cc:1261] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Refusing update from remote peer a9ea7335538d4aa9aa5409162f143ad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:50.494505 30940 consensus_queue.cc:1035] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:50.494810 30945 consensus_queue.cc:1035] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:50.528568 30458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:50.528854 30458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:50.532660 30458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Sending DeleteTablet for 3 replicas of tablet cc0d119a696c427e83e5b480907118cd
I20260619 16:52:50.533951 30850 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0d119a696c427e83e5b480907118cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:50 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:33682
I20260619 16:52:50.533988 30582 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0d119a696c427e83e5b480907118cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:50 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:38872
I20260619 16:52:50.534716 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 30519
I20260619 16:52:50.534734 30716 tablet_service.cc:1514] Processing DeleteTablet for tablet cc0d119a696c427e83e5b480907118cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:50 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:36770
I20260619 16:52:50.547271 30958 tablet_replica.cc:330] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: stopping tablet replica
I20260619 16:52:50.547894 30958 raft_consensus.cc:2229] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:50.548524 30958 raft_consensus.cc:2258] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:50.551985 30958 ts_tablet_manager.cc:1905] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:50.558964 30960 tablet_replica.cc:330] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: stopping tablet replica
I20260619 16:52:50.559664 30960 raft_consensus.cc:2229] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:52:50.560571 30960 raft_consensus.cc:2258] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:50.561214 30959 tablet_replica.cc:330] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: stopping tablet replica
I20260619 16:52:50.562008 30959 raft_consensus.cc:2229] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:50.564059 30960 ts_tablet_manager.cc:1905] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:52:50 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:52:50.564633 30959 raft_consensus.cc:2258] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:50.568812 30959 ts_tablet_manager.cc:1905] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:50.571143 30958 ts_tablet_manager.cc:1918] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:50.571333 30958 log.cc:1196] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/cc0d119a696c427e83e5b480907118cd
I20260619 16:52:50.571909 30958 ts_tablet_manager.cc:1939] T cc0d119a696c427e83e5b480907118cd P a3fe0621da7545ff89b52ff080526ea9: Deleting consensus metadata
I20260619 16:52:50.574044 30443 catalog_manager.cc:4841] TS a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727): tablet cc0d119a696c427e83e5b480907118cd (table test-table [id=289a3680b26d4f9caf6c1251fa75def7]) successfully deleted
I20260619 16:52:50.578902 30960 ts_tablet_manager.cc:1918] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:50.579149 30960 log.cc:1196] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/cc0d119a696c427e83e5b480907118cd
I20260619 16:52:50.579685 30960 ts_tablet_manager.cc:1939] T cc0d119a696c427e83e5b480907118cd P a9ea7335538d4aa9aa5409162f143ad0: Deleting consensus metadata
I20260619 16:52:50.580144 30959 ts_tablet_manager.cc:1918] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:50.580379 30959 log.cc:1196] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/cc0d119a696c427e83e5b480907118cd
I20260619 16:52:50.580981 30959 ts_tablet_manager.cc:1939] T cc0d119a696c427e83e5b480907118cd P 17881fb24f6f42a7a2092fd7266ca89e: Deleting consensus metadata
I20260619 16:52:50.581787 30445 catalog_manager.cc:4841] TS a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807): tablet cc0d119a696c427e83e5b480907118cd (table test-table [id=289a3680b26d4f9caf6c1251fa75def7]) successfully deleted
I20260619 16:52:50.583081 30445 catalog_manager.cc:4841] TS 17881fb24f6f42a7a2092fd7266ca89e (127.14.153.1:39035): tablet cc0d119a696c427e83e5b480907118cd (table test-table [id=289a3680b26d4f9caf6c1251fa75def7]) successfully deleted
I20260619 16:52:50.583320 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.536279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.14.153.1:39035)
0619 16:52:50.536517 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.536527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.546821 (+ 10294us) server_negotiation.cc:407] Connection header received
0619 16:52:50.547403 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.547410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.547499 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.547587 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:50.555209 (+ 7622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.556457 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.558715 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.559412 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.559561 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.574662 (+ 15101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:50.574682 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:50.574703 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:50.574759 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:50.579246 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.580396 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.580407 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.580411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.580600 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.581121 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.581127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.581130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.581628 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:50.581946 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.582626 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.582889 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:52:50.584009 30642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:50.733117 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 30519
I20260619 16:52:50.757321 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 30653
Jun 19 16:52:50 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:52:50.776144 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.758203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.14.153.2:42807)
0619 16:52:50.758401 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.758407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.758434 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:52:50.758634 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.758640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.758733 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.758857 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:50.760427 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.761567 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.763244 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.763930 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.764065 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.768906 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:50.768932 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:50.768953 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:50.769014 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:50.773046 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.773796 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.773807 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.773811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.773999 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.774370 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.774375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.774379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.774854 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:50.775211 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.775484 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.775724 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:52:50.777046 30776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:50.893035 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 30653
I20260619 16:52:50.912607 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 30787
Jun 19 16:52:50 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:52:50.932673 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:50.913453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.14.153.3:42727)
0619 16:52:50.913639 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:50.913644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:50.913662 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:52:50.913805 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:50.913809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:50.913873 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:50.913956 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:50.915365 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.916084 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.917461 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:50.917846 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:50.917944 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:50.927162 (+ 9218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:50.927178 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:50.927196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:50.927244 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:50.929849 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.930564 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.930571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.930574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.930718 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:50.931162 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:50.931166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:50.931169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:50.931570 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:50.931815 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:50.932206 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 16:52:50.932395 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:52:50.933370 30910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:51.045557 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 30787
I20260619 16:52:51.062250 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 30426
I20260619 16:52:51.063510 30488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:52:51.174124 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 30426
I20260619 16:53:11.196409 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42119
--webserver_interface=127.14.153.62
--webserver_port=38201
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:42119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:11.535307 30966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:11.535619 30966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:11.535725 30966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:11.543234 30966 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:53:11.543370 30966 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:53:11.543411 30966 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:53:11.543465 30966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:11.543522 30966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:11.543593 30966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:11.543648 30966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:11.543704 30966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:11.553997 30966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:42119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=38201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:11.555404 30966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:11.556751 30966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:11.580886 30972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:11.580824 30973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:11.581110 30966 server_base.cc:1034] running on GCE node
W20260619 16:53:11.580803 30975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:11.582301 30966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:11.584069 30966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:11.585314 30966 hybrid_clock.cc:648] HybridClock initialized: now 1781887991585238 us; error 50 us; skew 500 ppm
I20260619 16:53:11.585747 30966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:11 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887991, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:11.589664 30966 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:53:11.591588 30966 webserver.cc:460] Webserver started at http://127.14.153.62:38201/ using document root <none> and password file <none>
I20260619 16:53:11.592782 30966 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:11.592954 30966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:11.599375 30966 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:53:11.602672 30982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:11.604413 30966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:11.604647 30966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "0cf2eb17de13433fa0f96f14553c6092"
format_stamp: "Formatted at 2026-06-19 16:52:48 on dist-test-slave-qb36"
server_key: "44da3b9d2daf5d1628faa298f76d0b49"
server_key_iv: "43bc98210ea9984c41cdac3073cfbb92"
server_key_version: "encryptionkey@0"
I20260619 16:53:11.605783 30966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:11.619334 30966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:11.624078 30966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:11.624405 30966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:11.644364 30966 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:42119
I20260619 16:53:11.644397 31033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:42119 every 8 connection(s)
I20260619 16:53:11.646095 30966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:11.651721 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 30966
I20260619 16:53:11.652603 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:39035
--local_ip_for_outbound_sockets=127.14.153.1
--tserver_master_addrs=127.14.153.62:42119
--webserver_port=40789
--webserver_interface=127.14.153.1
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:53:11.653841 31034 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:53:11.657482 31034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Bootstrap starting.
I20260619 16:53:11.678066 31034 log.cc:824] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:11.690861 31034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:53:11.691468 31034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Bootstrap complete.
I20260619 16:53:11.700917 31034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:53:11.701809 31034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf2eb17de13433fa0f96f14553c6092, State: Initialized, Role: FOLLOWER
I20260619 16:53:11.702502 31034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:53:11.702752 31034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:11.702878 31034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:11.703083 31034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:53:11.707129 31034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:53:11.707707 31034 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf2eb17de13433fa0f96f14553c6092; no voters:
I20260619 16:53:11.708355 31034 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:53:11.708496 31038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:53:11.709246 31038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 2 LEADER]: Becoming Leader. State: Replica: 0cf2eb17de13433fa0f96f14553c6092, State: Running, Role: LEADER
I20260619 16:53:11.709873 31038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:53:11.711401 31034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:11.712460 31040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0cf2eb17de13433fa0f96f14553c6092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } } }
I20260619 16:53:11.712862 31040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:11.715270 31039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf2eb17de13433fa0f96f14553c6092. Latest consensus state: current_term: 2 leader_uuid: "0cf2eb17de13433fa0f96f14553c6092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } } }
I20260619 16:53:11.716069 31039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:11.717196 31044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:11.728070 31044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:11.730134 31044 catalog_manager.cc:1259] Loaded cluster ID: b257063516d54a04a160bfafce01bbe9
I20260619 16:53:11.730373 31044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:11.735860 31044 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:11.738158 31044 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Loaded TSK: 0
I20260619 16:53:11.779340 31055 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:53:11.795060 31044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Generated new TSK 1
I20260619 16:53:11.795499 31044 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:53:12.029583 31036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:12.029873 31036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:12.030011 31036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:12.037520 31036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:12.037642 31036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:12.037823 31036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:53:12.048126 31036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:39035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=40789
--tserver_master_addrs=127.14.153.62:42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:12.049518 31036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:12.050891 31036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:12.065559 31062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:12.065587 31061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:12.066421 31064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:12.066946 31036 server_base.cc:1034] running on GCE node
I20260619 16:53:12.067778 31036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:12.069029 31036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:12.070290 31036 hybrid_clock.cc:648] HybridClock initialized: now 1781887992070211 us; error 76 us; skew 500 ppm
I20260619 16:53:12.070698 31036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:12 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887992, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:12.074784 31036 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:53:12.076543 31036 webserver.cc:460] Webserver started at http://127.14.153.1:40789/ using document root <none> and password file <none>
I20260619 16:53:12.077375 31036 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:12.077527 31036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:12.083617 31036 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:53:12.086690 31071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:12.088400 31036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:12.088595 31036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "17881fb24f6f42a7a2092fd7266ca89e"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "af2e8f45e02bd439fb2184c81bb5ec03"
server_key_iv: "04a2c7484aced51d94954466ccfb264a"
server_key_version: "encryptionkey@0"
I20260619 16:53:12.089643 31036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:12.104532 31036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:12.108976 31036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:12.109277 31036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:12.110564 31036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:12.112213 31036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:12.112352 31036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:12.112476 31036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:12.112571 31036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:12.147917 31036 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:39035
I20260619 16:53:12.148011 31183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:39035 every 8 connection(s)
I20260619 16:53:12.149451 31036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:53:12.159173 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 31036
I20260619 16:53:12.159937 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:42807
--local_ip_for_outbound_sockets=127.14.153.2
--tserver_master_addrs=127.14.153.62:42119
--webserver_port=36383
--webserver_interface=127.14.153.2
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:53:12 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887992, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:53:12.175580 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:12.152545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:34245 (local address 127.14.153.62:42119)
0619 16:53:12.153284 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:12.153302 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:12.153832 (+ 530us) server_negotiation.cc:407] Connection header received
0619 16:53:12.156085 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:12.156099 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:12.156460 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:12.156852 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:12.158431 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:12.160492 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:12.162090 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:12.163225 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:12.163425 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:12.167567 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:12.167575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:12.167592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:12.167631 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:12.170678 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:12.171608 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:12.171616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:12.171622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:12.171795 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:12.172245 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:12.172250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:12.172257 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:12.172770 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:12.172977 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:12.173787 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 16:53:12.174162 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":320,"threads_started":1}
I20260619 16:53:12.175588 31187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:12.152774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:34245)
0619 16:53:12.153637 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:12.153676 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:12.155494 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:12.156995 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:12.157007 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:12.157666 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:12.158219 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:12.158236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:12.160674 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:12.160678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:12.161932 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:12.161945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:12.163434 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:12.163439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:12.163633 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:12.164367 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:12.164391 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:12.167273 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:12.170853 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:12.170863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:12.170880 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:12.171448 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:12.171922 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:12.171927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:12.171931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:12.172113 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:12.172925 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:12.172933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:12.173281 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 16:53:12.173561 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 16:53:12.174156 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":353,"threads_started":1}
I20260619 16:53:12.179281 31184 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42119
I20260619 16:53:12.179616 31184 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:12.180404 31184 heartbeater.cc:511] Master 127.14.153.62:42119 requested a full tablet report, sending...
I20260619 16:53:12.182740 30999 ts_manager.cc:194] Registered new tserver with Master: 17881fb24f6f42a7a2092fd7266ca89e (127.14.153.1:39035)
I20260619 16:53:12.184340 30999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:34245
W20260619 16:53:12.499236 31188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:12.499572 31188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:12.499747 31188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:12.507251 31188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:12.507414 31188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:12.507596 31188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:53:12.517740 31188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:42807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=36383
--tserver_master_addrs=127.14.153.62:42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:12.519235 31188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:12.520617 31188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:12.534430 31193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:12.534518 31196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:12.534456 31194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:12.535696 31188 server_base.cc:1034] running on GCE node
I20260619 16:53:12.536374 31188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:12.537400 31188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:12.538610 31188 hybrid_clock.cc:648] HybridClock initialized: now 1781887992538545 us; error 62 us; skew 500 ppm
I20260619 16:53:12.538965 31188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:12 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887992, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:12.542335 31188 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:53:12.543951 31188 webserver.cc:460] Webserver started at http://127.14.153.2:36383/ using document root <none> and password file <none>
I20260619 16:53:12.544682 31188 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:12.544847 31188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:12.550350 31188 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:53:12.553285 31203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:12.554627 31188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:53:12.554816 31188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "a9ea7335538d4aa9aa5409162f143ad0"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "c1e6e7fb71cddb5cb70ad9b852f862f1"
server_key_iv: "8c3a5cdb6ca33d9a477e7a6f884e51e6"
server_key_version: "encryptionkey@0"
I20260619 16:53:12.555913 31188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:12.578349 31188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:12.583257 31188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:12.583547 31188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:12.584599 31188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:12.586021 31188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:12.586145 31188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:12.586275 31188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:12.586372 31188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:12.621815 31188 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:42807
I20260619 16:53:12.621850 31315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:42807 every 8 connection(s)
I20260619 16:53:12.623354 31188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:12.626538 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 31188
I20260619 16:53:12.627249 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:42727
--local_ip_for_outbound_sockets=127.14.153.3
--tserver_master_addrs=127.14.153.62:42119
--webserver_port=41971
--webserver_interface=127.14.153.3
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:53:12 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887992, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:53:12.646721 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:12.626069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:56807 (local address 127.14.153.62:42119)
0619 16:53:12.626336 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:12.626344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:12.627660 (+ 1316us) server_negotiation.cc:407] Connection header received
0619 16:53:12.629746 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:12.629755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:12.629858 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:12.629977 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:12.631568 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:12.632709 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:12.634282 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:12.635086 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:12.635237 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:12.639119 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:12.639129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:12.639139 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:12.639183 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:12.642788 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:12.643685 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:12.643694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:12.643700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:12.643857 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:12.644365 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:12.644374 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:12.644380 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:12.644818 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:12.645026 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:12.645898 (+ 872us) server_negotiation.cc:299] Negotiation successful
0619 16:53:12.646212 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 16:53:12.647771 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:12.626561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:56807)
0619 16:53:12.627481 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:12.627533 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:12.629451 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:12.630182 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:12.630191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:12.630708 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:12.631383 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:12.631403 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:12.632874 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:12.632877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:12.634119 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:12.634131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:12.635325 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:12.635330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:12.635535 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:12.636290 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:12.636319 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:12.638821 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:12.642947 (+ 4126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:12.642954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:12.642967 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:12.643543 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:12.643982 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:12.643987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:12.643990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:12.644219 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:12.645015 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:12.645027 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:12.645437 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 16:53:12.645741 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 16:53:12.646304 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":401,"threads_started":1}
I20260619 16:53:12.649652 31316 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42119
I20260619 16:53:12.650038 31316 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:12.651113 31316 heartbeater.cc:511] Master 127.14.153.62:42119 requested a full tablet report, sending...
I20260619 16:53:12.653048 30999 ts_manager.cc:194] Registered new tserver with Master: a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807)
I20260619 16:53:12.654232 30999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:56807
W20260619 16:53:12.972193 31319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:12.972519 31319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:12.972692 31319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:12.980214 31319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:12.980378 31319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:12.980559 31319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:53:12.990561 31319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:42727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=41971
--tserver_master_addrs=127.14.153.62:42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:12.992054 31319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:12.993571 31319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:13.007753 31327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:13.007864 31325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:13.007856 31324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:13.008921 31319 server_base.cc:1034] running on GCE node
I20260619 16:53:13.009653 31319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:13.010718 31319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:13.011936 31319 hybrid_clock.cc:648] HybridClock initialized: now 1781887993011858 us; error 59 us; skew 500 ppm
I20260619 16:53:13.012310 31319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:13 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887993, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:13.015745 31319 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:53:13.017323 31319 webserver.cc:460] Webserver started at http://127.14.153.3:41971/ using document root <none> and password file <none>
I20260619 16:53:13.018066 31319 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:13.018246 31319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:13.024055 31319 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 16:53:13.026957 31334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:13.028472 31319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:53:13.028664 31319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "a3fe0621da7545ff89b52ff080526ea9"
format_stamp: "Formatted at 2026-06-19 16:52:50 on dist-test-slave-qb36"
server_key: "401f7d0c0c8076949cecadebda8267f3"
server_key_iv: "7f7c472c5aa054bce3ee1d9a8fa87eed"
server_key_version: "encryptionkey@0"
I20260619 16:53:13.029696 31319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:13.051913 31319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:13.056401 31319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:13.056708 31319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:13.057744 31319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:13.059288 31319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:13.059402 31319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:13.059547 31319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:13.059609 31319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:13.093796 31319 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:42727
I20260619 16:53:13.093914 31446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:42727 every 8 connection(s)
I20260619 16:53:13.095495 31319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:13.096611 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 31319
Jun 19 16:53:13 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887993, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:53:13.113137 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.097861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.14.153.62:42119)
0619 16:53:13.098114 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.098122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.098562 (+ 440us) server_negotiation.cc:407] Connection header received
0619 16:53:13.098798 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.098802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.098867 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.098980 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:13.100788 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.101507 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.103068 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.103503 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.103650 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.106473 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:13.106478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:13.106481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:13.106513 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:13.109844 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.110609 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.110616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.110620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.110765 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.111240 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.111245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.111257 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.111666 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:13.111964 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.112396 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.112674 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 16:53:13.117403 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.098418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:45895 (local address 127.14.153.62:42119)
0619 16:53:13.099414 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.099419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.100140 (+ 721us) server_negotiation.cc:407] Connection header received
0619 16:53:13.102001 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.102007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.102081 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.102192 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:13.103857 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.104805 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.106379 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.106879 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.107025 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.110742 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:13.110747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:13.110751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:13.110782 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:13.114105 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.114988 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.114993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.114997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.115151 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.115606 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.115611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.115614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.115898 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:13.116057 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.116772 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.116993 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":748,"threads_started":1}
I20260619 16:53:13.118427 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.099128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:45895)
0619 16:53:13.099978 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.100024 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:13.101755 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:13.102464 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:13.102472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:13.102988 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:13.103659 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.103678 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.104968 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.104974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.106218 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.106230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.107164 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.107172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.107431 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.108021 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:13.108047 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:13.110506 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:13.114247 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:13.114257 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:13.114275 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:13.114855 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:13.115269 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:13.115275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:13.115278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:13.115478 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:13.116033 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:13.116041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:13.116410 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 16:53:13.116687 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 16:53:13.117251 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":353,"threads_started":1}
I20260619 16:53:13.119912 31447 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42119
I20260619 16:53:13.120242 31447 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:13.120993 31447 heartbeater.cc:511] Master 127.14.153.62:42119 requested a full tablet report, sending...
I20260619 16:53:13.122529 30999 ts_manager.cc:194] Registered new tserver with Master: a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727)
I20260619 16:53:13.123529 30999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:45895
I20260619 16:53:13.128394 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:53:13.135579 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.129529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.14.153.62:42119)
0619 16:53:13.129750 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.129755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.130178 (+ 423us) server_negotiation.cc:407] Connection header received
0619 16:53:13.130355 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.130359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.130509 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.130641 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:13.131198 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.131934 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.133519 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.133921 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.134057 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.134651 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:13.134883 (+ 232us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 16:53:13.135304 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53180: token expired
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 16:53:13.135743 31450 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53180: token expired
W20260619 16:53:13.136214 30919 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.153.62:42119)
I20260619 16:53:13.149936 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.136725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.14.153.62:42119)
0619 16:53:13.136917 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.136923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.137035 (+ 112us) server_negotiation.cc:407] Connection header received
0619 16:53:13.137204 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.137209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.137283 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.137402 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:13.138815 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.139622 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.141051 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.141466 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.141607 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.144399 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:13.144410 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:13.144414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:13.144447 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:13.146907 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.147744 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.147751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.147755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.147895 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.148377 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.148382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.148385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.148829 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:13.149092 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.149345 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.149584 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:53:13.153980 30999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:53:13.161557 30999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:53:13.187949 31184 heartbeater.cc:503] Master 127.14.153.62:42119 was elected leader, sending a full tablet report...
I20260619 16:53:13.200139 31459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.173527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.0.0.1:44238)
0619 16:53:13.186601 (+ 13074us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.186675 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:13.186956 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:13.191395 (+ 4439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:13.191403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:13.191455 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:13.192238 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.192248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.193999 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.194007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.196785 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.196799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.198657 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.198666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.198890 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.198944 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:53:13.199103 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 16:53:13.199338 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12636,"mutex_wait_us":296,"spinlock_wait_cycles":9728,"thread_start_us":14899,"threads_started":1}
I20260619 16:53:13.200299 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.173899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.14.153.1:39035)
0619 16:53:13.190139 (+ 16240us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.190151 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.190188 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:53:13.190313 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.190324 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.190819 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.191069 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:13.192426 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.193865 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.196975 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.198402 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.198552 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.198818 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.199747 (+ 929us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.199952 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16012,"thread_start_us":16541,"threads_started":1}
I20260619 16:53:13.200484 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.174218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.14.153.3:42727)
0619 16:53:13.188754 (+ 14536us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.188769 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.188817 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:53:13.190684 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.190692 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.190918 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.191124 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:13.192351 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.194062 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.197272 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.198522 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.198866 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.199147 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.199911 (+ 764us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.200113 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14281,"thread_start_us":14130,"threads_started":1}
I20260619 16:53:13.200886 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.189068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:42807 (local address 127.0.0.1:37438)
0619 16:53:13.191234 (+ 2166us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.191250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:13.191364 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:13.192152 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:13.192162 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:13.192190 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:13.192663 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.192677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.194730 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.194736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.197342 (+ 2606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.197357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.199558 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.199566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.199756 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.199787 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:53:13.199871 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:53:13.200004 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2040,"mutex_wait_us":6,"thread_start_us":1876,"threads_started":1}
I20260619 16:53:13.201251 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.173896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:42727 (local address 127.0.0.1:50250)
0619 16:53:13.188190 (+ 14294us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.188221 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:13.188356 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:13.191397 (+ 3041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:13.191403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:13.191453 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:13.192057 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.192074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.194227 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.194232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.197041 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.197056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.199313 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.199320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.199510 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.199547 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:53:13.199655 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 16:53:13.199801 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14169,"mutex_wait_us":104,"spinlock_wait_cycles":8960,"thread_start_us":13396,"threads_started":1}
I20260619 16:53:13.201637 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.189136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.14.153.2:42807)
0619 16:53:13.190726 (+ 1590us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.190738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.191497 (+ 759us) server_negotiation.cc:407] Connection header received
0619 16:53:13.191605 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.191612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.191875 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.192083 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:13.192853 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.194345 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.197514 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.198968 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.199180 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.199383 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.200294 (+ 911us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.201249 (+ 955us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1368,"thread_start_us":1233,"threads_started":1}
I20260619 16:53:13.204763 31382 tablet_service.cc:1467] Processing CreateTablet for tablet 108f60c05460490babcc2c907a8049b9 (DEFAULT_TABLE table=test-table [id=4046741052b8477886bc8e7338e28b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:13.204921 31251 tablet_service.cc:1467] Processing CreateTablet for tablet 108f60c05460490babcc2c907a8049b9 (DEFAULT_TABLE table=test-table [id=4046741052b8477886bc8e7338e28b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:13.204909 31119 tablet_service.cc:1467] Processing CreateTablet for tablet 108f60c05460490babcc2c907a8049b9 (DEFAULT_TABLE table=test-table [id=4046741052b8477886bc8e7338e28b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:13.206126 31382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108f60c05460490babcc2c907a8049b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:13.206255 31251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108f60c05460490babcc2c907a8049b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:13.206274 31119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108f60c05460490babcc2c907a8049b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:13.220129 31463 tablet_bootstrap.cc:492] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: Bootstrap starting.
I20260619 16:53:13.222062 31464 tablet_bootstrap.cc:492] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Bootstrap starting.
I20260619 16:53:13.223634 31465 tablet_bootstrap.cc:492] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: Bootstrap starting.
I20260619 16:53:13.224231 31463 tablet_bootstrap.cc:654] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:13.225633 31463 log.cc:824] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:13.225780 31464 tablet_bootstrap.cc:654] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:13.226653 31465 tablet_bootstrap.cc:654] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:13.227245 31464 log.cc:824] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:13.227878 31465 log.cc:824] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:13.230811 31464 tablet_bootstrap.cc:492] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: No bootstrap required, opened a new log
I20260619 16:53:13.231253 31464 ts_tablet_manager.cc:1397] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 16:53:13.231302 31465 tablet_bootstrap.cc:492] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: No bootstrap required, opened a new log
I20260619 16:53:13.231554 31465 ts_tablet_manager.cc:1397] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260619 16:53:13.233392 31463 tablet_bootstrap.cc:492] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: No bootstrap required, opened a new log
I20260619 16:53:13.233743 31463 ts_tablet_manager.cc:1397] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260619 16:53:13.241494 31463 raft_consensus.cc:348] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:13.241936 31463 raft_consensus.cc:374] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:13.242086 31463 raft_consensus.cc:729] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fe0621da7545ff89b52ff080526ea9, State: Initialized, Role: FOLLOWER
I20260619 16:53:13.242760 31463 consensus_queue.cc:260] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:13.242584 31465 raft_consensus.cc:348] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:13.242583 31464 raft_consensus.cc:348] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:13.243062 31464 raft_consensus.cc:374] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:13.243081 31465 raft_consensus.cc:374] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:13.243180 31465 raft_consensus.cc:729] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ea7335538d4aa9aa5409162f143ad0, State: Initialized, Role: FOLLOWER
I20260619 16:53:13.243234 31464 raft_consensus.cc:729] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17881fb24f6f42a7a2092fd7266ca89e, State: Initialized, Role: FOLLOWER
I20260619 16:53:13.244007 31447 heartbeater.cc:503] Master 127.14.153.62:42119 was elected leader, sending a full tablet report...
I20260619 16:53:13.243966 31464 consensus_queue.cc:260] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:13.244753 31463 ts_tablet_manager.cc:1428] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 16:53:13.245827 31464 ts_tablet_manager.cc:1428] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260619 16:53:13.246942 31465 consensus_queue.cc:260] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:13.248370 31316 heartbeater.cc:503] Master 127.14.153.62:42119 was elected leader, sending a full tablet report...
I20260619 16:53:13.249872 31465 ts_tablet_manager.cc:1428] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
W20260619 16:53:13.347445 31448 tablet.cc:2369] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:13.375491 31317 tablet.cc:2369] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:13.401942 31185 tablet.cc:2369] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:13.402375 31469 raft_consensus.cc:482] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:13.402678 31469 raft_consensus.cc:504] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:13.404474 31469 leader_election.cc:288] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17881fb24f6f42a7a2092fd7266ca89e (127.14.153.1:39035), a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807)
I20260619 16:53:13.410537 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.404951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.14.153.3:42423)
0619 16:53:13.405238 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.405262 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:13.405420 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:13.406048 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:13.406053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:13.406086 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:13.406268 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.406275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.407382 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.407390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.408683 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.408694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.409812 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.409817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.409994 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.410018 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:53:13.410128 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:53:13.410257 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":1}
I20260619 16:53:13.410534 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.405050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:42423 (local address 127.14.153.1:39035)
0619 16:53:13.405301 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.405310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.405333 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:53:13.405622 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.405630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.405725 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.405865 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:13.406417 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.407247 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.408846 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.409645 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.409812 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.409885 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.410155 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.410279 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:53:13.411563 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.405423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:42807 (local address 127.14.153.3:40283)
0619 16:53:13.406138 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.406153 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:13.406315 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:13.406638 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:13.406642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:13.406675 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:13.406994 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.407003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.407996 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.408001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.409542 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:13.409557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.410935 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.410941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:13.411129 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.411160 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:53:13.411250 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:53:13.411353 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":433,"threads_started":1}
I20260619 16:53:13.411731 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.405775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:40283 (local address 127.14.153.2:42807)
0619 16:53:13.406003 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.406010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.406244 (+ 234us) server_negotiation.cc:407] Connection header received
0619 16:53:13.406465 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.406470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.406527 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.406625 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:13.407184 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.407856 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.409709 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.410685 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.410842 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.410903 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.411319 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.411451 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 16:53:13.411733 31139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108f60c05460490babcc2c907a8049b9" candidate_uuid: "a3fe0621da7545ff89b52ff080526ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17881fb24f6f42a7a2092fd7266ca89e" is_pre_election: true
I20260619 16:53:13.412408 31139 raft_consensus.cc:2454] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fe0621da7545ff89b52ff080526ea9 in term 0.
I20260619 16:53:13.412727 31271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108f60c05460490babcc2c907a8049b9" candidate_uuid: "a3fe0621da7545ff89b52ff080526ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ea7335538d4aa9aa5409162f143ad0" is_pre_election: true
I20260619 16:53:13.413121 31338 leader_election.cc:302] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17881fb24f6f42a7a2092fd7266ca89e, a3fe0621da7545ff89b52ff080526ea9; no voters:
I20260619 16:53:13.413314 31271 raft_consensus.cc:2454] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fe0621da7545ff89b52ff080526ea9 in term 0.
I20260619 16:53:13.413529 31469 raft_consensus.cc:2781] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:53:13.413692 31469 raft_consensus.cc:482] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:53:13.413820 31469 raft_consensus.cc:3037] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:13.416172 31469 raft_consensus.cc:504] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:13.416821 31469 leader_election.cc:288] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 17881fb24f6f42a7a2092fd7266ca89e (127.14.153.1:39035), a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807)
I20260619 16:53:13.417299 31139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108f60c05460490babcc2c907a8049b9" candidate_uuid: "a3fe0621da7545ff89b52ff080526ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17881fb24f6f42a7a2092fd7266ca89e"
I20260619 16:53:13.417423 31271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108f60c05460490babcc2c907a8049b9" candidate_uuid: "a3fe0621da7545ff89b52ff080526ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ea7335538d4aa9aa5409162f143ad0"
I20260619 16:53:13.417510 31139 raft_consensus.cc:3037] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:13.417699 31271 raft_consensus.cc:3037] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:13.419924 31139 raft_consensus.cc:2454] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fe0621da7545ff89b52ff080526ea9 in term 1.
I20260619 16:53:13.420327 31271 raft_consensus.cc:2454] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fe0621da7545ff89b52ff080526ea9 in term 1.
I20260619 16:53:13.420346 31338 leader_election.cc:302] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17881fb24f6f42a7a2092fd7266ca89e, a3fe0621da7545ff89b52ff080526ea9; no voters:
I20260619 16:53:13.420662 31469 raft_consensus.cc:2781] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:13.421164 31469 raft_consensus.cc:686] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 1 LEADER]: Becoming Leader. State: Replica: a3fe0621da7545ff89b52ff080526ea9, State: Running, Role: LEADER
I20260619 16:53:13.421746 31469 consensus_queue.cc:237] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:13.427127 30997 catalog_manager.cc:5495] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3). New cstate: current_term: 1 leader_uuid: "a3fe0621da7545ff89b52ff080526ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } health_report { overall_health: HEALTHY } } }
I20260619 16:53:13.446375 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.441423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.14.153.3:42727)
0619 16:53:13.441601 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.441606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.441706 (+ 100us) server_negotiation.cc:407] Connection header received
0619 16:53:13.441872 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.441875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.441952 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.442078 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:13.442376 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.443185 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.444496 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.444878 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.444963 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.445415 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:13.445920 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:13.446035 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.446099 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.446164 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:53:13.454403 31271 raft_consensus.cc:1261] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [term 1 FOLLOWER]: Refusing update from remote peer a3fe0621da7545ff89b52ff080526ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:13.454414 31139 raft_consensus.cc:1261] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Refusing update from remote peer a3fe0621da7545ff89b52ff080526ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:13.455401 31473 consensus_queue.cc:1035] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:13.455888 31469 consensus_queue.cc:1035] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:13.486826 30998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:13.487212 30998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:13.491272 30998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Sending DeleteTablet for 3 replicas of tablet 108f60c05460490babcc2c907a8049b9
I20260619 16:53:13.492594 31251 tablet_service.cc:1514] Processing DeleteTablet for tablet 108f60c05460490babcc2c907a8049b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:13 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:37438
I20260619 16:53:13.492594 31382 tablet_service.cc:1514] Processing DeleteTablet for tablet 108f60c05460490babcc2c907a8049b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:13 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:50250
I20260619 16:53:13.492566 31119 tablet_service.cc:1514] Processing DeleteTablet for tablet 108f60c05460490babcc2c907a8049b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:13 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:44238
I20260619 16:53:13.493561 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 31036
I20260619 16:53:13.499078 31486 tablet_replica.cc:330] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: stopping tablet replica
I20260619 16:53:13.499404 31487 tablet_replica.cc:330] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: stopping tablet replica
I20260619 16:53:13.500265 31486 raft_consensus.cc:2229] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:13.500871 31486 raft_consensus.cc:2258] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:13.501754 31488 tablet_replica.cc:330] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: stopping tablet replica
I20260619 16:53:13.502125 31487 raft_consensus.cc:2229] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:13.502562 31487 raft_consensus.cc:2258] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:13.503360 31488 raft_consensus.cc:2229] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:53:13.504148 31488 raft_consensus.cc:2258] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:13.505947 31486 ts_tablet_manager.cc:1905] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:13.506583 31488 ts_tablet_manager.cc:1905] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:13.507989 31487 ts_tablet_manager.cc:1905] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:13.513468 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.495505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.14.153.1:39035)
0619 16:53:13.495754 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.495764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.495791 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:53:13.495861 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.495865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.495966 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.496089 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:13.498369 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.499673 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.501948 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.502626 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.502775 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.506206 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:13.506217 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:13.506236 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:13.506285 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:13.510231 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.511070 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.511080 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.511083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.511271 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.511694 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.511699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.511720 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.512220 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:13.512585 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.512847 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.513074 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:53:13.514415 31179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:13.515615 31488 ts_tablet_manager.cc:1918] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:13.515925 31488 log.cc:1196] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/108f60c05460490babcc2c907a8049b9
I20260619 16:53:13.516690 31488 ts_tablet_manager.cc:1939] T 108f60c05460490babcc2c907a8049b9 P a3fe0621da7545ff89b52ff080526ea9: Deleting consensus metadata
I20260619 16:53:13.518936 31487 ts_tablet_manager.cc:1918] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:13.519152 31487 log.cc:1196] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/108f60c05460490babcc2c907a8049b9
I20260619 16:53:13.519168 30984 catalog_manager.cc:4841] TS a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727): tablet 108f60c05460490babcc2c907a8049b9 (table test-table [id=4046741052b8477886bc8e7338e28b12]) successfully deleted
I20260619 16:53:13.519739 31487 ts_tablet_manager.cc:1939] T 108f60c05460490babcc2c907a8049b9 P a9ea7335538d4aa9aa5409162f143ad0: Deleting consensus metadata
I20260619 16:53:13.521653 30986 catalog_manager.cc:4841] TS a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807): tablet 108f60c05460490babcc2c907a8049b9 (table test-table [id=4046741052b8477886bc8e7338e28b12]) successfully deleted
I20260619 16:53:13.613771 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 31036
I20260619 16:53:13.635776 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 31188
W20260619 16:53:13.635788 30986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.153.1:39035 (error 108)
I20260619 16:53:13.638520 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.637978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.0.0.1:44266)
0619 16:53:13.638223 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.638342 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:39035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260619 16:53:13.638976 30986 catalog_manager.cc:4584] TS 17881fb24f6f42a7a2092fd7266ca89e (127.14.153.1:39035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 108f60c05460490babcc2c907a8049b9: Network error: Client connection negotiation failed: client connection to 127.14.153.1:39035: connect: Connection refused (error 111)
I20260619 16:53:13.658722 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.637443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.14.153.2:42807)
0619 16:53:13.637640 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.637648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.637668 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:53:13.637731 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.637736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.637850 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.637953 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:13.639928 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.640806 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.642228 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.642653 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.642801 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.650825 (+ 8024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:13.650840 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:13.650858 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:13.650903 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:13.654971 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.656091 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.656102 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.656106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.656270 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.656687 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.656693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.656696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.657219 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:13.657493 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.658096 (+ 603us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.658358 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:53:13.659675 31311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:13.692478 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.691995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.0.0.1:44276)
0619 16:53:13.692221 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.692324 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:39035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:53:13.693601 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.693249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.0.0.1:44280)
0619 16:53:13.693396 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.693483 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:39035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:53:13.754449 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.753846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.0.0.1:44294)
0619 16:53:13.754122 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.754243 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:39035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 16:53:13.755782 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.755357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.0.0.1:44306)
0619 16:53:13.755524 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.755616 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:39035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:53:13.788062 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 31188
I20260619 16:53:13.805560 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 31319
I20260619 16:53:13.820885 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.806420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.14.153.3:42727)
0619 16:53:13.806636 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:13.806642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:13.806714 (+ 72us) server_negotiation.cc:407] Connection header received
0619 16:53:13.806916 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:13.806922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:13.807011 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:13.807150 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:13.808429 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.809271 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.810677 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:13.811376 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:13.811530 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:13.814488 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:13.814500 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:13.814517 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:13.814560 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:13.817670 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.818537 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.818547 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.818550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.818726 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:13.819158 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:13.819163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:13.819170 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:13.819624 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:13.819983 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:13.820275 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 16:53:13.820499 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:53:13.821620 31442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:13.837070 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.836544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.0.0.1:44308)
0619 16:53:13.836775 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.836888 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:39035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 16:53:13.838240 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.837849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:39035 (local address 127.0.0.1:44324)
0619 16:53:13.838055 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.838137 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:39035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 16:53:13.946527 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 31319
I20260619 16:53:13.966327 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 30966
I20260619 16:53:13.967628 31029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:14.084424 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 30966
I20260619 16:53:34.106858 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42119
--webserver_interface=127.14.153.62
--webserver_port=38201
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:42119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:34.447136 31494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:34.447521 31494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:34.447726 31494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:34.455135 31494 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:53:34.455272 31494 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:53:34.455370 31494 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:53:34.455495 31494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:34.455588 31494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:34.455695 31494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:34.455785 31494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:34.455861 31494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:34.466079 31494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:42119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=38201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:34.467551 31494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:34.468904 31494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:34.480908 31499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:34.480957 31500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:34.480865 31502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:34.487398 31494 server_base.cc:1034] running on GCE node
I20260619 16:53:34.488240 31494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:34.489701 31494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:34.490918 31494 hybrid_clock.cc:648] HybridClock initialized: now 1781888014490845 us; error 63 us; skew 500 ppm
I20260619 16:53:34.491288 31494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:34 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888014, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:34.494663 31494 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:53:34.496388 31494 webserver.cc:460] Webserver started at http://127.14.153.62:38201/ using document root <none> and password file <none>
I20260619 16:53:34.497165 31494 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:34.497301 31494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:34.504014 31494 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:53:34.507014 31509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:34.508467 31494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:34.508692 31494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "0cf2eb17de13433fa0f96f14553c6092"
format_stamp: "Formatted at 2026-06-19 16:52:48 on dist-test-slave-qb36"
server_key: "44da3b9d2daf5d1628faa298f76d0b49"
server_key_iv: "43bc98210ea9984c41cdac3073cfbb92"
server_key_version: "encryptionkey@0"
I20260619 16:53:34.509673 31494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:34.531244 31494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:34.535584 31494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:34.535876 31494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:34.555909 31494 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:42119
I20260619 16:53:34.555932 31560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:42119 every 8 connection(s)
I20260619 16:53:34.557482 31494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:34.561704 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 31494
I20260619 16:53:34.562757 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:39035
--local_ip_for_outbound_sockets=127.14.153.1
--tserver_master_addrs=127.14.153.62:42119
--webserver_port=40789
--webserver_interface=127.14.153.1
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:53:34.566399 31561 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:53:34.572065 31561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Bootstrap starting.
I20260619 16:53:34.593255 31561 log.cc:824] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:34.609441 31561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:53:34.610049 31561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Bootstrap complete.
I20260619 16:53:34.619896 31561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:53:34.620806 31561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf2eb17de13433fa0f96f14553c6092, State: Initialized, Role: FOLLOWER
I20260619 16:53:34.621444 31561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:53:34.621691 31561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:34.621809 31561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:34.622025 31561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:53:34.625872 31561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:53:34.626359 31561 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf2eb17de13433fa0f96f14553c6092; no voters:
I20260619 16:53:34.627074 31561 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:53:34.627221 31565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:53:34.629412 31561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:34.630115 31565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [term 3 LEADER]: Becoming Leader. State: Replica: 0cf2eb17de13433fa0f96f14553c6092, State: Running, Role: LEADER
I20260619 16:53:34.630743 31565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } }
I20260619 16:53:34.633709 31567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf2eb17de13433fa0f96f14553c6092. Latest consensus state: current_term: 3 leader_uuid: "0cf2eb17de13433fa0f96f14553c6092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } } }
I20260619 16:53:34.634044 31567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:34.634579 31566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0cf2eb17de13433fa0f96f14553c6092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf2eb17de13433fa0f96f14553c6092" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42119 } } }
I20260619 16:53:34.634876 31566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:34.643414 31575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:34.652187 31575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:34.654326 31575 catalog_manager.cc:1259] Loaded cluster ID: b257063516d54a04a160bfafce01bbe9
I20260619 16:53:34.654546 31575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:34.658921 31582 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:53:34.659888 31575 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:34.661079 31575 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Loaded TSK: 0
I20260619 16:53:34.661221 31575 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0cf2eb17de13433fa0f96f14553c6092: Loaded TSK: 1
F20260619 16:53:34.662050 31575 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5f751fb9ad google::LogMessage::Fail() at ??:0
@ 0x7f5f751fe88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5f751fb4de google::LogMessage::Flush() at ??:0
@ 0x7f5f751fca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5f78f5f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5f78f4e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5f8ac2a1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5f8ac29783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5f8acb52a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f5f8acb50c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f5f8acff5e9 std::function<>::operator()() at ??:0
@ 0x7f5f8ac2dcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5f8ac2bc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5f8acb3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5f8acb3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5f77e47345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5f77e4a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5f77e4a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5f77e25eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5f7b0ff6db start_thread at ??:0
@ 0x7f5f7409561f clone at ??:0
W20260619 16:53:34.940807 31563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:34.941144 31563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:34.941326 31563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:34.948909 31563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:34.949075 31563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:34.949257 31563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:53:34.959349 31563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:39035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=40789
--tserver_master_addrs=127.14.153.62:42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:34.960690 31563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:34.962020 31563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:34.975251 31609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:34.975257 31611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:34.975324 31608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:34.976231 31563 server_base.cc:1034] running on GCE node
I20260619 16:53:34.976878 31563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:34.977759 31563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:34.978956 31563 hybrid_clock.cc:648] HybridClock initialized: now 1781888014978919 us; error 34 us; skew 500 ppm
I20260619 16:53:34.979323 31563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:34 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888014, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:34.982448 31563 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:53:34.983914 31563 webserver.cc:460] Webserver started at http://127.14.153.1:40789/ using document root <none> and password file <none>
I20260619 16:53:34.984654 31563 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:34.984758 31563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:34.990108 31563 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:53:34.993026 31618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:34.994364 31563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:53:34.994490 31563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "17881fb24f6f42a7a2092fd7266ca89e"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "af2e8f45e02bd439fb2184c81bb5ec03"
server_key_iv: "04a2c7484aced51d94954466ccfb264a"
server_key_version: "encryptionkey@0"
I20260619 16:53:34.995503 31563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:35.012449 31563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:35.016628 31563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:35.016867 31563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:35.017809 31563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:35.019743 31625 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 16:53:35.025566 31563 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:53:35.025717 31563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260619 16:53:35.025815 31563 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:53:35.029129 31563 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:53:35.029273 31563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.001s
I20260619 16:53:35.029610 31625 tablet_bootstrap.cc:492] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Bootstrap starting.
I20260619 16:53:35.038769 31625 log.cc:824] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:35.055586 31625 tablet_bootstrap.cc:492] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:53:35.056284 31625 tablet_bootstrap.cc:492] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Bootstrap complete.
I20260619 16:53:35.056771 31625 ts_tablet_manager.cc:1397] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.008s
I20260619 16:53:35.066546 31625 raft_consensus.cc:348] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:35.068315 31625 raft_consensus.cc:729] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17881fb24f6f42a7a2092fd7266ca89e, State: Initialized, Role: FOLLOWER
I20260619 16:53:35.069205 31625 consensus_queue.cc:260] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:35.071271 31625 ts_tablet_manager.cc:1428] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260619 16:53:35.074687 31563 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:39035
I20260619 16:53:35.074783 31733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:39035 every 8 connection(s)
I20260619 16:53:35.076074 31563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:53:35.080708 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 31563
I20260619 16:53:35.081540 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:42807
--local_ip_for_outbound_sockets=127.14.153.2
--tserver_master_addrs=127.14.153.62:42119
--webserver_port=36383
--webserver_interface=127.14.153.2
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:53:35.081600 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.079144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:33777)
0619 16:53:35.079948 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:35.080282 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":333,"threads_started":1}
W20260619 16:53:35.082288 31734 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:42119 (0 consecutive failures): Network error: Failed to ping master at 127.14.153.62:42119: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
I20260619 16:53:35.083338 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.082924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:54043)
0619 16:53:35.083159 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:35.083232 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 16:53:35.084501 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.084056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:60923)
0619 16:53:35.084275 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:35.084356 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:53:35.084812 31734 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:53:35.417598 31737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:35.417899 31737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:35.417995 31737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:35.425302 31737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:35.425420 31737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:35.425534 31737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:53:35.435442 31737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:42807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=36383
--tserver_master_addrs=127.14.153.62:42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:35.436764 31737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:35.438115 31737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:35.451179 31742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:35.452329 31743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:35.453181 31745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:35.453547 31737 server_base.cc:1034] running on GCE node
I20260619 16:53:35.454175 31737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:35.455264 31737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:35.456488 31737 hybrid_clock.cc:648] HybridClock initialized: now 1781888015456443 us; error 57 us; skew 500 ppm
I20260619 16:53:35.456821 31737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:35 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888015, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:35.459978 31737 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:53:35.461350 31737 webserver.cc:460] Webserver started at http://127.14.153.2:36383/ using document root <none> and password file <none>
I20260619 16:53:35.462020 31737 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:35.462118 31737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:35.467676 31737 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:53:35.470459 31752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:35.471866 31737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:53:35.472043 31737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "a9ea7335538d4aa9aa5409162f143ad0"
format_stamp: "Formatted at 2026-06-19 16:52:49 on dist-test-slave-qb36"
server_key: "c1e6e7fb71cddb5cb70ad9b852f862f1"
server_key_iv: "8c3a5cdb6ca33d9a477e7a6f884e51e6"
server_key_version: "encryptionkey@0"
I20260619 16:53:35.473032 31737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:35.489034 31737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:35.493568 31737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:35.493840 31737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:35.495070 31737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:35.497010 31737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:35.497130 31737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:35.497246 31737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:35.497337 31737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:35.533310 31737 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:42807
I20260619 16:53:35.533396 31864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:42807 every 8 connection(s)
I20260619 16:53:35.534854 31737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:35.536140 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 31737
I20260619 16:53:35.536770 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:42727
--local_ip_for_outbound_sockets=127.14.153.3
--tserver_master_addrs=127.14.153.62:42119
--webserver_port=41971
--webserver_interface=127.14.153.3
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:53:35.540267 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.538309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:51483)
0619 16:53:35.539169 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:35.539399 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":329,"threads_started":1}
W20260619 16:53:35.541005 31865 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:42119 (0 consecutive failures): Network error: Failed to ping master at 127.14.153.62:42119: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
I20260619 16:53:35.541929 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.541628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:57989)
0619 16:53:35.541764 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:35.541825 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 16:53:35.542791 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.542445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:43037)
0619 16:53:35.542592 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:35.542688 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:53:35.543013 31865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:53:35.869043 31868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:35.869325 31868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:35.869518 31868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:35.877017 31868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:35.877166 31868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:35.877329 31868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:53:35.887885 31868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:42727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=41971
--tserver_master_addrs=127.14.153.62:42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:35.889174 31868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:35.890501 31868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:35.903966 31873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:35.903939 31874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:35.904151 31876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:35.904644 31868 server_base.cc:1034] running on GCE node
I20260619 16:53:35.905215 31868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:35.906092 31868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:35.907313 31868 hybrid_clock.cc:648] HybridClock initialized: now 1781888015907254 us; error 56 us; skew 500 ppm
I20260619 16:53:35.907625 31868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:35 dist-test-slave-qb36 krb5kdc[30413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888015, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:35.910595 31868 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:53:35.912056 31868 webserver.cc:460] Webserver started at http://127.14.153.3:41971/ using document root <none> and password file <none>
I20260619 16:53:35.912750 31868 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:35.912899 31868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:35.918308 31868 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 16:53:35.921183 31883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:35.922603 31868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:53:35.922791 31868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "a3fe0621da7545ff89b52ff080526ea9"
format_stamp: "Formatted at 2026-06-19 16:52:50 on dist-test-slave-qb36"
server_key: "401f7d0c0c8076949cecadebda8267f3"
server_key_iv: "7f7c472c5aa054bce3ee1d9a8fa87eed"
server_key_version: "encryptionkey@0"
I20260619 16:53:35.923847 31868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:35.950562 31868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:35.954587 31868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:35.954883 31868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:35.955938 31868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:35.957333 31868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:35.957458 31868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:35.957609 31868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:35.957703 31868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:35.992136 31868 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:42727
I20260619 16:53:35.992187 31995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:42727 every 8 connection(s)
I20260619 16:53:35.993574 31868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:35.998206 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.996438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:38213)
0619 16:53:35.997150 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:35.997392 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":272,"threads_started":1}
W20260619 16:53:35.998896 31996 heartbeater.cc:644] Failed to heartbeat to 127.14.153.62:42119 (0 consecutive failures): Network error: Failed to ping master at 127.14.153.62:42119: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
I20260619 16:53:35.999670 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.999388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:55997)
0619 16:53:35.999511 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:35.999573 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:53:36.000473 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:36.000194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:54447)
0619 16:53:36.000300 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:36.000367 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:53:36.000694 31996 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:53:36.001595 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 31868
I20260619 16:53:36.086026 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:36.085298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:47799)
0619 16:53:36.085737 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:36.085818 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260619 16:53:36.352073 32001 raft_consensus.cc:482] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:36.352391 32001 raft_consensus.cc:504] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:36.354020 32001 leader_election.cc:288] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807), a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727)
Jun 19 16:53:36 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888014, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
Jun 19 16:53:36 dist-test-slave-qb36 krb5kdc[30413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888014, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:53:36.380355 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:36.354261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:42807 (local address 127.14.153.1:38531)
0619 16:53:36.354597 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:36.354650 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:36.355374 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:36.359130 (+ 3756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:36.359142 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:36.359999 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:36.360757 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:36.360776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.362395 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.362400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:36.364135 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:36.364147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.365169 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.365177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:36.365478 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:36.369260 (+ 3782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:36.369278 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:36.371722 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:36.375707 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:36.375717 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:36.375733 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:36.376399 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:36.377025 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:36.377028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:36.377030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:36.377306 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:36.378347 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:36.378356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:36.378880 (+ 524us) client_negotiation.cc:769] Sending connection context
0619 16:53:36.379278 (+ 398us) client_negotiation.cc:240] Negotiation successful
0619 16:53:36.379965 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":456,"spinlock_wait_cycles":33280}
I20260619 16:53:36.380425 32004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:36.354780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:41459 (local address 127.14.153.3:42727)
0619 16:53:36.356980 (+ 2200us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:36.356991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:36.357023 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:53:36.358412 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:36.358420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:36.358755 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:36.359124 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:36.360976 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.362162 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.364309 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.364919 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.365110 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:36.372343 (+ 7233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:36.372350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:36.372360 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:36.372395 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:36.375909 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:36.376785 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:36.376792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:36.376799 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:36.376997 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:36.377431 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:36.377436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:36.377440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:36.378151 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:36.378361 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:36.379582 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0619 16:53:36.379994 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2058,"thread_start_us":1921,"threads_started":1}
I20260619 16:53:36.380446 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:36.354487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:38531 (local address 127.14.153.2:42807)
0619 16:53:36.356662 (+ 2175us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:36.356685 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:36.356736 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:53:36.358224 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:36.358233 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:36.358627 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:36.358891 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:36.360959 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.362167 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.364275 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.364922 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.365134 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:36.372105 (+ 6971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:36.372119 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:36.372140 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:36.372193 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:36.375550 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:36.376612 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:36.376622 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:36.376635 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:36.376904 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:36.377431 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:36.377436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:36.377440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:36.378151 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:36.378390 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:36.379570 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0619 16:53:36.379994 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1933,"thread_start_us":1820,"threads_started":1}
I20260619 16:53:36.380362 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:36.354565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:42727 (local address 127.14.153.1:41459)
0619 16:53:36.356329 (+ 1764us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:36.356346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:36.356448 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:36.359130 (+ 2682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:36.359142 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:36.359998 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:36.360764 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:36.360776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.362363 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.362373 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:36.364141 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:36.364153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.365146 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.365153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:36.365484 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:36.369219 (+ 3735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:36.369243 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:36.372082 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:36.376051 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:36.376056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:36.376061 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:36.376623 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:36.377110 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:36.377114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:36.377117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:36.377306 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:36.378347 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:36.378356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:36.378880 (+ 524us) client_negotiation.cc:769] Sending connection context
0619 16:53:36.379278 (+ 398us) client_negotiation.cc:240] Negotiation successful
0619 16:53:36.379846 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1601,"mutex_wait_us":319,"spinlock_wait_cycles":31488,"thread_start_us":1348,"threads_started":1}
W20260619 16:53:36.384011 31622 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
W20260619 16:53:36.384261 31620 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
I20260619 16:53:36.384456 31620 leader_election.cc:302] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17881fb24f6f42a7a2092fd7266ca89e; no voters: a3fe0621da7545ff89b52ff080526ea9, a9ea7335538d4aa9aa5409162f143ad0
I20260619 16:53:36.384883 32001 raft_consensus.cc:2726] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:36.544350 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:36.543507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:37413)
0619 16:53:36.543959 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:36.544122 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":229,"threads_started":1}
I20260619 16:53:37.002264 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:37.001444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:49077)
0619 16:53:37.001914 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:37.002041 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":241,"threads_started":1}
I20260619 16:53:37.087803 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:37.086976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:41553)
0619 16:53:37.087525 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:37.087622 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":221,"threads_started":1}
I20260619 16:53:37.545941 32008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:37.545256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:49129)
0619 16:53:37.545698 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:37.545784 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":229,"threads_started":1}
I20260619 16:53:38.004065 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:38.003352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:36427)
0619 16:53:38.003807 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:38.003891 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":232,"threads_started":1}
I20260619 16:53:38.089447 32010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:38.088727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:36841)
0619 16:53:38.089193 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:38.089275 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260619 16:53:38.387544 32011 raft_consensus.cc:482] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:38.387768 32011 raft_consensus.cc:504] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:38.388536 32011 leader_election.cc:288] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807), a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727)
W20260619 16:53:38.389541 31622 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
W20260619 16:53:38.389755 31620 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
I20260619 16:53:38.389868 31620 leader_election.cc:302] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17881fb24f6f42a7a2092fd7266ca89e; no voters: a3fe0621da7545ff89b52ff080526ea9, a9ea7335538d4aa9aa5409162f143ad0
I20260619 16:53:38.390148 32011 raft_consensus.cc:2726] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:38.547794 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:38.546935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:36767)
0619 16:53:38.547550 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:38.547635 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":362,"threads_started":1}
I20260619 16:53:39.005517 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:39.004886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:37681)
0619 16:53:39.005279 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:39.005361 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":209,"threads_started":1}
I20260619 16:53:39.091245 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:39.090394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:59475)
0619 16:53:39.090904 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:39.090992 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":303,"threads_started":1}
I20260619 16:53:39.549608 32015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:39.548786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:44719)
0619 16:53:39.549271 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:39.549362 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260619 16:53:40.007258 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:40.006481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:50193)
0619 16:53:40.006965 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:40.007079 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":259,"threads_started":1}
I20260619 16:53:40.093135 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:40.092341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:33961)
0619 16:53:40.092857 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:40.092950 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":254,"threads_started":1}
I20260619 16:53:40.551697 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:40.550724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:58933)
0619 16:53:40.551400 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:40.551501 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":338,"threads_started":1}
I20260619 16:53:41.008903 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:41.008212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:45313)
0619 16:53:41.008658 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:41.008750 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":218,"threads_started":1}
I20260619 16:53:41.094935 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:41.094116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:50641)
0619 16:53:41.094653 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:41.094758 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":255,"threads_started":1}
I20260619 16:53:41.425858 32021 raft_consensus.cc:482] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:41.426074 32021 raft_consensus.cc:504] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:41.426788 32021 leader_election.cc:288] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807), a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727)
W20260619 16:53:41.427820 31622 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
W20260619 16:53:41.428043 31620 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
I20260619 16:53:41.428206 31620 leader_election.cc:302] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17881fb24f6f42a7a2092fd7266ca89e; no voters: a3fe0621da7545ff89b52ff080526ea9, a9ea7335538d4aa9aa5409162f143ad0
I20260619 16:53:41.428478 32021 raft_consensus.cc:2726] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:41.553278 32022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:41.552576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:47589)
0619 16:53:41.553019 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:41.553101 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260619 16:53:42.010468 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:42.009749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:48807)
0619 16:53:42.010204 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:42.010285 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
I20260619 16:53:42.096817 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:42.095935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:44603)
0619 16:53:42.096461 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:42.096595 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":245,"threads_started":1}
I20260619 16:53:42.555187 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:42.554228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:42795)
0619 16:53:42.554818 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:42.554907 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":292,"threads_started":1}
I20260619 16:53:43.012260 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:43.011437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:55127)
0619 16:53:43.011955 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:43.012037 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":247,"threads_started":1}
I20260619 16:53:43.098431 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:43.097719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:43857)
0619 16:53:43.098187 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:43.098268 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
I20260619 16:53:43.556869 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:43.556171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:45509)
0619 16:53:43.556589 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:43.556680 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260619 16:53:44.013908 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:44.013212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:46301)
0619 16:53:44.013638 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:44.013724 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20260619 16:53:44.100278 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:44.099542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:54329)
0619 16:53:44.100004 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:44.100087 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260619 16:53:44.558525 32031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:44.557769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:40645)
0619 16:53:44.558257 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:44.558340 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
I20260619 16:53:44.864423 32032 raft_consensus.cc:482] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:44.864603 32032 raft_consensus.cc:504] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:44.865245 32032 leader_election.cc:288] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807), a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727)
W20260619 16:53:44.866250 31622 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
W20260619 16:53:44.866448 31620 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
I20260619 16:53:44.866653 31620 leader_election.cc:302] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17881fb24f6f42a7a2092fd7266ca89e; no voters: a3fe0621da7545ff89b52ff080526ea9, a9ea7335538d4aa9aa5409162f143ad0
I20260619 16:53:44.866891 32032 raft_consensus.cc:2726] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:45.015709 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:45.014922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:44507)
0619 16:53:45.015434 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:45.015538 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":278,"threads_started":1}
I20260619 16:53:45.102203 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:45.101377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:37587)
0619 16:53:45.101875 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:45.102003 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":238,"threads_started":1}
I20260619 16:53:45.560463 32035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:45.559642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:59275)
0619 16:53:45.560200 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:45.560284 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":348,"threads_started":1}
I20260619 16:53:46.017401 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:46.016626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:54557)
0619 16:53:46.017122 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:46.017209 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":278,"threads_started":1}
I20260619 16:53:46.103909 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:46.103192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:40895)
0619 16:53:46.103642 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:46.103725 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260619 16:53:46.562274 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:46.561479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:50061)
0619 16:53:46.562003 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:46.562090 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":287,"threads_started":1}
I20260619 16:53:47.019220 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:47.018419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:37929)
0619 16:53:47.018910 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:47.018994 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":261,"threads_started":1}
I20260619 16:53:47.105845 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:47.105007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:53595)
0619 16:53:47.105549 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:47.105666 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":316,"threads_started":1}
I20260619 16:53:47.564157 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:47.563389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:60057)
0619 16:53:47.563892 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:47.563978 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":279,"threads_started":1}
I20260619 16:53:48.021028 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.020300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:54515)
0619 16:53:48.020775 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.020861 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":229,"threads_started":1}
I20260619 16:53:48.107632 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.106784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:58105)
0619 16:53:48.107347 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.107449 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":333,"threads_started":1}
I20260619 16:53:48.566005 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.565143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:40031)
0619 16:53:48.565619 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.565722 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
I20260619 16:53:49.022910 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:49.022065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:35505)
0619 16:53:49.022635 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:49.022752 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":310,"threads_started":1}
I20260619 16:53:49.109305 32046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:49.108564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:46521)
0619 16:53:49.109022 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:49.109122 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
I20260619 16:53:49.568006 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:49.567104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:44361)
0619 16:53:49.567632 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:49.567798 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":250,"threads_started":1}
I20260619 16:53:50.024840 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:50.023879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:33875)
0619 16:53:50.024468 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:50.024623 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":314,"threads_started":1}
I20260619 16:53:50.111471 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:50.110669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.1:35513)
0619 16:53:50.111206 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:50.111301 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":242,"threads_started":1}
I20260619 16:53:50.124101 32050 raft_consensus.cc:482] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:50.124365 32050 raft_consensus.cc:504] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17881fb24f6f42a7a2092fd7266ca89e" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 39035 } } peers { permanent_uuid: "a9ea7335538d4aa9aa5409162f143ad0" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 42807 } } peers { permanent_uuid: "a3fe0621da7545ff89b52ff080526ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42727 } }
I20260619 16:53:50.125310 32050 leader_election.cc:288] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807), a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727)
W20260619 16:53:50.126456 31620 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fe0621da7545ff89b52ff080526ea9 (127.14.153.3:42727): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
W20260619 16:53:50.126691 31622 leader_election.cc:341] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9ea7335538d4aa9aa5409162f143ad0 (127.14.153.2:42807): Not found: Tablet not found: 108f60c05460490babcc2c907a8049b9
I20260619 16:53:50.126838 31622 leader_election.cc:302] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17881fb24f6f42a7a2092fd7266ca89e; no voters: a3fe0621da7545ff89b52ff080526ea9, a9ea7335538d4aa9aa5409162f143ad0
I20260619 16:53:50.127206 32050 raft_consensus.cc:2726] T 108f60c05460490babcc2c907a8049b9 P 17881fb24f6f42a7a2092fd7266ca89e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:50.569931 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:50.569111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.2:59613)
0619 16:53:50.569639 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:50.569744 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":255,"threads_started":1}
I20260619 16:53:51.002935 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 31563
I20260619 16:53:51.021836 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.004146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.14.153.1:39035)
0619 16:53:51.005007 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.005018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.005054 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:53:51.005135 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.005146 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.005473 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.005665 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.007146 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.008112 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.009286 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.009829 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.009926 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.016023 (+ 6097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.016038 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.016054 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.016096 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.018834 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.019671 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.019680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.019683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.019838 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.020192 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.020195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.020198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.020630 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.020884 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.021235 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.021451 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":581,"threads_started":1}
I20260619 16:53:51.023144 31729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:51.027081 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.026098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42119 (local address 127.14.153.3:50031)
0619 16:53:51.026637 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.026776 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.62:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":233,"threads_started":1}
I20260619 16:53:51.115775 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 31563
I20260619 16:53:51.133401 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 31737
I20260619 16:53:51.152405 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.134215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.14.153.2:42807)
0619 16:53:51.134822 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.134826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.134843 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:53:51.134898 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.134901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.134961 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.135112 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.136493 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.137366 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.138618 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.139201 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.139302 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.146758 (+ 7456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.146765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.146769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.146812 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.149479 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.150238 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.150242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.150244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.150379 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.150792 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.150796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.150798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.151272 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.151470 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.151796 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.151976 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":370,"threads_started":1}
I20260619 16:53:51.153198 31860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:51.238306 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 31737
I20260619 16:53:51.257984 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 31868
I20260619 16:53:51.271481 32058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.258808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.14.153.3:42727)
0619 16:53:51.259406 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.259410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.259427 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:53:51.259482 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.259485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.259555 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.259650 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.261018 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.261818 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.262991 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.263481 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.263579 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.266119 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.266123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.266126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.266155 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.269034 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.269727 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.269731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.269733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.269865 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.270222 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.270225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.270227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.270487 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.270720 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.270955 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.271188 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":347,"threads_started":1}
I20260619 16:53:51.272195 31991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:51.376979 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 31868
2026-06-19T16:53:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64626 ms)
[----------] 30 tests from SecurityITest (296399 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:51 dist-test-slave-qb36 krb5kdc[32062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:51 dist-test-slave-qb36 krb5kdc[32062](info): set up 2 sockets
Jun 19 16:53:51 dist-test-slave-qb36 krb5kdc[32062](info): commencing operation
krb5kdc: starting...
W20260619 16:53:52.491456 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:53:52 dist-test-slave-qb36 krb5kdc[32062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:53:52.743188 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:41153
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:41153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:53.086642 32075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:53.086953 32075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:53.087162 32075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:53.094587 32075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:53.094707 32075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:53.094820 32075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:53.094920 32075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:53.094985 32075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:53.105096 32075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:41153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:53.106469 32075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:53.107878 32075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:53.120438 32081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:53.120360 32083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:53.120473 32080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:53.120846 32075 server_base.cc:1034] running on GCE node
I20260619 16:53:53.121846 32075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:53.123373 32075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:53.124634 32075 hybrid_clock.cc:648] HybridClock initialized: now 1781888033124561 us; error 36 us; skew 500 ppm
I20260619 16:53:53.125134 32075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:53 dist-test-slave-qb36 krb5kdc[32062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:53.128460 32075 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:53:53.129981 32075 webserver.cc:460] Webserver started at http://127.14.153.62:32971/ using document root <none> and password file <none>
I20260619 16:53:53.130719 32075 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:53.130867 32075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:53.131248 32075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:53.134325 32075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "71917f3d1911443db1aad9ae077718ae"
format_stamp: "Formatted at 2026-06-19 16:53:53 on dist-test-slave-qb36"
server_key: "476121b8fa852dff5f52cca567b4fe24"
server_key_iv: "c082f2267ba390229da6e36f60b6819a"
server_key_version: "encryptionkey@0"
I20260619 16:53:53.135167 32075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "71917f3d1911443db1aad9ae077718ae"
format_stamp: "Formatted at 2026-06-19 16:53:53 on dist-test-slave-qb36"
server_key: "476121b8fa852dff5f52cca567b4fe24"
server_key_iv: "c082f2267ba390229da6e36f60b6819a"
server_key_version: "encryptionkey@0"
I20260619 16:53:53.140573 32075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:53:53.144119 32090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:53.145478 32075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:53:53.145702 32075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "71917f3d1911443db1aad9ae077718ae"
format_stamp: "Formatted at 2026-06-19 16:53:53 on dist-test-slave-qb36"
server_key: "476121b8fa852dff5f52cca567b4fe24"
server_key_iv: "c082f2267ba390229da6e36f60b6819a"
server_key_version: "encryptionkey@0"
I20260619 16:53:53.145916 32075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:53.180172 32075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:53.184769 32075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:53.185077 32075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:53.204295 32075 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:41153
I20260619 16:53:53.204303 32141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:41153 every 8 connection(s)
I20260619 16:53:53.205790 32075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:53.207634 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 32075
I20260619 16:53:53.207868 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:53:53.208712 14948 external_mini_cluster.cc:1428] Setting key 6d4b0b92d0af07d57578e68f4d9ed40e
I20260619 16:53:53.210819 32142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 16:53:53 dist-test-slave-qb36 krb5kdc[32062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:53:53.225248 32142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae: Bootstrap starting.
I20260619 16:53:53.228933 32142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:53.230003 32142 log.cc:824] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:53.233034 32142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae: No bootstrap required, opened a new log
I20260619 16:53:53.233570 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:53.210926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.14.153.62:41153)
0619 16:53:53.211798 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:53.211809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:53.211851 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:53:53.213467 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:53.213489 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:53.213911 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:53.214282 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:53.216241 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.217832 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.219829 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.220542 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.220714 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:53.225952 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:53.225961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:53.225977 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:53.226022 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:53.229361 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:53.230108 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:53.230116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:53.230123 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:53.230319 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:53.230726 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:53.230731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:53.230734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:53.231353 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:53.231688 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:53.232106 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 16:53:53.232530 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":370,"threads_started":1}
I20260619 16:53:53.241178 32142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71917f3d1911443db1aad9ae077718ae" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41153 } }
I20260619 16:53:53.241580 32142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:53.241690 32142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71917f3d1911443db1aad9ae077718ae, State: Initialized, Role: FOLLOWER
I20260619 16:53:53.242278 32142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71917f3d1911443db1aad9ae077718ae" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41153 } }
I20260619 16:53:53.242515 32142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:53.242656 32142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:53.242831 32142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:53.245441 32142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71917f3d1911443db1aad9ae077718ae" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41153 } }
I20260619 16:53:53.245962 32142 leader_election.cc:302] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71917f3d1911443db1aad9ae077718ae; no voters:
I20260619 16:53:53.246569 32142 leader_election.cc:288] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:53.246706 32147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:53.247457 32147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [term 1 LEADER]: Becoming Leader. State: Replica: 71917f3d1911443db1aad9ae077718ae, State: Running, Role: LEADER
I20260619 16:53:53.247988 32147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71917f3d1911443db1aad9ae077718ae" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41153 } }
I20260619 16:53:53.248782 32142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:53.251116 32149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71917f3d1911443db1aad9ae077718ae. Latest consensus state: current_term: 1 leader_uuid: "71917f3d1911443db1aad9ae077718ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71917f3d1911443db1aad9ae077718ae" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41153 } } }
I20260619 16:53:53.251530 32149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:53.254194 32154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:53.254047 32148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71917f3d1911443db1aad9ae077718ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71917f3d1911443db1aad9ae077718ae" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 41153 } } }
I20260619 16:53:53.254366 32148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:53.259708 32154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:53.271816 32154 catalog_manager.cc:1347] Generated new cluster ID: d81e9a910d5345c791d05539d92bf532
I20260619 16:53:53.271938 32154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:53.296586 32154 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:53.297977 32154 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:53.304755 32154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae: Generated new TSK 0
I20260619 16:53:53.305596 32154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:53:53.368834 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41153
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:53.703385 32168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:53.703688 32168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:53.703787 32168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:53.710943 32168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:53.711095 32168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:53.711220 32168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:53:53.721573 32168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:53.723093 32168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:53.724515 32168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:53.738664 32173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:53.738759 32174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:53.739073 32176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:53.740268 32168 server_base.cc:1034] running on GCE node
I20260619 16:53:53.740921 32168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:53.741948 32168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:53.743146 32168 hybrid_clock.cc:648] HybridClock initialized: now 1781888033743107 us; error 49 us; skew 500 ppm
I20260619 16:53:53.743487 32168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:53 dist-test-slave-qb36 krb5kdc[32062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:53.746733 32168 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:53:53.748308 32168 webserver.cc:460] Webserver started at http://127.14.153.1:42801/ using document root <none> and password file <none>
I20260619 16:53:53.749090 32168 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:53.749277 32168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:53.749635 32168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:53.752772 32168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01"
format_stamp: "Formatted at 2026-06-19 16:53:53 on dist-test-slave-qb36"
server_key: "d66de86d68b0c112ea76f519ee6604e6"
server_key_iv: "8a3afd41a92078358e26654269c7ff11"
server_key_version: "encryptionkey@0"
I20260619 16:53:53.753527 32168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01"
format_stamp: "Formatted at 2026-06-19 16:53:53 on dist-test-slave-qb36"
server_key: "d66de86d68b0c112ea76f519ee6604e6"
server_key_iv: "8a3afd41a92078358e26654269c7ff11"
server_key_version: "encryptionkey@0"
I20260619 16:53:53.759606 32168 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 16:53:53.763324 32183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:53.764779 32168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:53:53.764962 32168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01"
format_stamp: "Formatted at 2026-06-19 16:53:53 on dist-test-slave-qb36"
server_key: "d66de86d68b0c112ea76f519ee6604e6"
server_key_iv: "8a3afd41a92078358e26654269c7ff11"
server_key_version: "encryptionkey@0"
I20260619 16:53:53.765182 32168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:53.799784 32168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:53.805560 32168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:53.805922 32168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:53.807188 32168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:53.808580 32168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:53.808691 32168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:53.808812 32168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:53.808902 32168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:53.840754 32168 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:43133
I20260619 16:53:53.840801 32295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:43133 every 8 connection(s)
I20260619 16:53:53.842150 32168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:53:53.846971 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 32168
I20260619 16:53:53.847214 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:53:53.847848 14948 external_mini_cluster.cc:1428] Setting key fc47c247429aeb38c05cdf33c44c2ecc
Jun 19 16:53:53 dist-test-slave-qb36 krb5kdc[32062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:53:53.864405 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:53.844919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:45107 (local address 127.14.153.62:41153)
0619 16:53:53.845411 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:53.845418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:53.846374 (+ 956us) server_negotiation.cc:407] Connection header received
0619 16:53:53.848400 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:53.848407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:53.848499 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:53.848633 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:53.850418 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.851639 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.853251 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.853959 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.854112 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:53.858015 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:53.858023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:53.858029 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:53.858076 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:53.860855 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:53.861863 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:53.861867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:53.861869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:53.861996 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:53.862527 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:53.862530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:53.862532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:53.862840 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:53.862959 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:53.863959 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0619 16:53:53.864150 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":228,"threads_started":1}
I20260619 16:53:53.865990 32299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:53.845516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41153 (local address 127.14.153.1:45107)
0619 16:53:53.846210 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:53.846249 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:53.848115 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:53.848924 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:53.848939 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:53.849514 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:53.850218 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:53.850239 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.851802 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.851806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:53.853090 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:53.853102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.854170 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.854175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:53.854389 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:53.855162 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:53.855188 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:53.857751 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:53.861026 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:53.861039 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:53.861063 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:53.861740 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:53.862111 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:53.862119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:53.862125 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:53.862417 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:53.862969 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:53.862981 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:53.863507 (+ 526us) client_negotiation.cc:769] Sending connection context
0619 16:53:53.863841 (+ 334us) client_negotiation.cc:240] Negotiation successful
0619 16:53:53.864457 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":271,"threads_started":1}
I20260619 16:53:53.867826 32296 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41153
I20260619 16:53:53.868150 32296 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:53.868958 32296 heartbeater.cc:511] Master 127.14.153.62:41153 requested a full tablet report, sending...
I20260619 16:53:53.871515 32107 ts_manager.cc:194] Registered new tserver with Master: ce04b3fbe0ed476e9db8c1ff68c9ae01 (127.14.153.1:43133)
I20260619 16:53:53.873929 32107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:45107
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:53:53.906427 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41153
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:54.238695 32302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:54.238987 32302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:54.239182 32302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:54.246608 32302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:54.246721 32302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:54.246938 32302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:53:54.257454 32302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:54.258778 32302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:54.260123 32302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:54.273397 32307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:54.273490 32308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:54.273440 32310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:54.276613 32302 server_base.cc:1034] running on GCE node
I20260619 16:53:54.277274 32302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:54.278323 32302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:54.279567 32302 hybrid_clock.cc:648] HybridClock initialized: now 1781888034279517 us; error 44 us; skew 500 ppm
I20260619 16:53:54.279924 32302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:54 dist-test-slave-qb36 krb5kdc[32062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:54.283226 32302 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:53:54.284843 32302 webserver.cc:460] Webserver started at http://127.14.153.2:42705/ using document root <none> and password file <none>
I20260619 16:53:54.285571 32302 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:54.285710 32302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:54.286036 32302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:54.288967 32302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "43e1df91dd5f422f9445ec34a930689e"
format_stamp: "Formatted at 2026-06-19 16:53:54 on dist-test-slave-qb36"
server_key: "5a1f4d774d2c736c393518aae8213a1e"
server_key_iv: "4848877bcfa280dc6ba9f6a50efca226"
server_key_version: "encryptionkey@0"
I20260619 16:53:54.289741 32302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "43e1df91dd5f422f9445ec34a930689e"
format_stamp: "Formatted at 2026-06-19 16:53:54 on dist-test-slave-qb36"
server_key: "5a1f4d774d2c736c393518aae8213a1e"
server_key_iv: "4848877bcfa280dc6ba9f6a50efca226"
server_key_version: "encryptionkey@0"
I20260619 16:53:54.295125 32302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:53:54.298645 32317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:54.300027 32302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:53:54.300220 32302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "43e1df91dd5f422f9445ec34a930689e"
format_stamp: "Formatted at 2026-06-19 16:53:54 on dist-test-slave-qb36"
server_key: "5a1f4d774d2c736c393518aae8213a1e"
server_key_iv: "4848877bcfa280dc6ba9f6a50efca226"
server_key_version: "encryptionkey@0"
I20260619 16:53:54.300426 32302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:54.322069 32302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:54.326167 32302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:54.326445 32302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:54.327464 32302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:54.328809 32302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:54.328915 32302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:54.329052 32302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:54.329126 32302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:54.362421 32302 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:43709
I20260619 16:53:54.362473 32429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:43709 every 8 connection(s)
I20260619 16:53:54.363781 32302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:54.371299 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 32302
I20260619 16:53:54.371490 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:53:54.371964 14948 external_mini_cluster.cc:1428] Setting key 7035675d67065946131f3280c20b1034
Jun 19 16:53:54 dist-test-slave-qb36 krb5kdc[32062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:53:54.386881 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.366217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:45749 (local address 127.14.153.62:41153)
0619 16:53:54.366699 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:54.366703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:54.367690 (+ 987us) server_negotiation.cc:407] Connection header received
0619 16:53:54.369646 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:54.369651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:54.369722 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:54.369832 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:54.371680 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.372535 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.374391 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.374841 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.374935 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.379436 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:54.379446 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:54.379453 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:54.379507 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:54.383255 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:54.384399 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:54.384404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:54.384406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:54.384547 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:54.385060 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:54.385063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:54.385065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:54.385341 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:54.385449 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:54.386403 (+ 954us) server_negotiation.cc:299] Negotiation successful
0619 16:53:54.386594 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":273,"threads_started":1}
I20260619 16:53:54.388523 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.366789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41153 (local address 127.14.153.2:45749)
0619 16:53:54.367519 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:54.367558 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:54.369380 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:54.370050 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:54.370059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:54.370653 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:54.371455 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.371484 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.372721 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.372727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.374243 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.374258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.375082 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.375093 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.375431 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.376206 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:54.376243 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:54.379138 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:54.383421 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:54.383433 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:54.383457 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:54.384236 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:54.384667 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:54.384675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:54.384682 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:54.384948 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:54.385460 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:54.385472 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:54.385975 (+ 503us) client_negotiation.cc:769] Sending connection context
0619 16:53:54.386299 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 16:53:54.386935 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":303,"threads_started":1}
I20260619 16:53:54.390059 32430 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41153
I20260619 16:53:54.390384 32430 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:54.391166 32430 heartbeater.cc:511] Master 127.14.153.62:41153 requested a full tablet report, sending...
I20260619 16:53:54.392616 32107 ts_manager.cc:194] Registered new tserver with Master: 43e1df91dd5f422f9445ec34a930689e (127.14.153.2:43709)
I20260619 16:53:54.393442 32107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:45749
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:53:54.430370 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41153
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:54.763761 32436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:54.764104 32436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:54.764292 32436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:54.771814 32436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:54.771968 32436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:54.772133 32436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:53:54.782233 32436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:54.783744 32436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:54.785100 32436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:54.802877 32444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:54.802992 32442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:54.802937 32441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:54.804867 32436 server_base.cc:1034] running on GCE node
I20260619 16:53:54.805629 32436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:54.806631 32436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:54.807852 32436 hybrid_clock.cc:648] HybridClock initialized: now 1781888034807778 us; error 61 us; skew 500 ppm
I20260619 16:53:54.808213 32436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:53:54 dist-test-slave-qb36 krb5kdc[32062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:54.811586 32436 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:53:54.813062 32436 webserver.cc:460] Webserver started at http://127.14.153.3:36679/ using document root <none> and password file <none>
I20260619 16:53:54.813849 32436 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:54.813998 32436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:54.814333 32436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:54.817376 32436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "9613bdc21e8540c7af2e17ece248a98f"
format_stamp: "Formatted at 2026-06-19 16:53:54 on dist-test-slave-qb36"
server_key: "72fa9f32e87b36846f45618a825c7710"
server_key_iv: "82a0a5f5c04ed735080cfe4fc8d87136"
server_key_version: "encryptionkey@0"
I20260619 16:53:54.818159 32436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "9613bdc21e8540c7af2e17ece248a98f"
format_stamp: "Formatted at 2026-06-19 16:53:54 on dist-test-slave-qb36"
server_key: "72fa9f32e87b36846f45618a825c7710"
server_key_iv: "82a0a5f5c04ed735080cfe4fc8d87136"
server_key_version: "encryptionkey@0"
I20260619 16:53:54.823518 32436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:53:54.827008 32451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:54.828454 32436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:53:54.828637 32436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "9613bdc21e8540c7af2e17ece248a98f"
format_stamp: "Formatted at 2026-06-19 16:53:54 on dist-test-slave-qb36"
server_key: "72fa9f32e87b36846f45618a825c7710"
server_key_iv: "82a0a5f5c04ed735080cfe4fc8d87136"
server_key_version: "encryptionkey@0"
I20260619 16:53:54.828840 32436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:54.865268 32436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:54.869729 32436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:54.870011 32436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:54.870981 32436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:54.872315 32436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:54.872386 32436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:54.872474 32436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:54.872519 32436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:54.877641 32296 heartbeater.cc:503] Master 127.14.153.62:41153 was elected leader, sending a full tablet report...
I20260619 16:53:54.906648 32436 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:36253
I20260619 16:53:54.906714 32563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:36253 every 8 connection(s)
I20260619 16:53:54.908025 32436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:54.914726 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 32436
I20260619 16:53:54.914923 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:53:54.915395 14948 external_mini_cluster.cc:1428] Setting key 58d0b518c2511cae456f4ba0a8765d3a
Jun 19 16:53:54 dist-test-slave-qb36 krb5kdc[32062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:53:54.928525 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.910627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:42099 (local address 127.14.153.62:41153)
0619 16:53:54.911109 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:54.911116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:54.912257 (+ 1141us) server_negotiation.cc:407] Connection header received
0619 16:53:54.914216 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:54.914225 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:54.914353 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:54.914485 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:54.916286 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.917217 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.918977 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.919453 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.919591 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.922981 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:54.922988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:54.922992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:54.923092 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:54.925471 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:54.926166 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:54.926172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:54.926175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:54.926305 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:54.926719 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:54.926725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:54.926728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:54.927002 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:54.927202 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:54.927897 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 16:53:54.928137 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":198,"threads_started":1}
I20260619 16:53:54.929564 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.911333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:41153 (local address 127.14.153.3:42099)
0619 16:53:54.912071 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:54.912109 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:54.913905 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:54.914787 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:54.914816 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:54.915476 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:54.916104 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.916123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.917366 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.917371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.918829 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.918842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.919680 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.919687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.919916 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.920565 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:54.920594 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:54.922781 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:54.925608 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:54.925617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:54.925635 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:54.926044 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:54.926410 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:54.926415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:54.926419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:54.926611 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:54.927162 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:54.927171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:54.927530 (+ 359us) client_negotiation.cc:769] Sending connection context
0619 16:53:54.927797 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 16:53:54.928333 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260619 16:53:54.931073 32564 heartbeater.cc:347] Connected to a master server at 127.14.153.62:41153
I20260619 16:53:54.931413 32564 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:54.932210 32564 heartbeater.cc:511] Master 127.14.153.62:41153 requested a full tablet report, sending...
I20260619 16:53:54.933724 32106 ts_manager.cc:194] Registered new tserver with Master: 9613bdc21e8540c7af2e17ece248a98f (127.14.153.3:36253)
I20260619 16:53:54.934648 32106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:42099
I20260619 16:53:54.943742 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:53:54.961293 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.948243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.14.153.62:41153)
0619 16:53:54.948403 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:54.948424 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:54.948906 (+ 482us) server_negotiation.cc:407] Connection header received
0619 16:53:54.949106 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:54.949110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:54.949178 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:54.949280 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:54.950800 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.951604 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.952907 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.953316 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.953448 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.955919 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:54.955924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:54.955927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:54.955959 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:54.958482 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:54.959217 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:54.959223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:54.959231 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:54.959352 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:54.959737 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:54.959744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:54.959747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:54.960075 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:54.960335 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:54.960724 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 16:53:54.960957 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:53:54.965345 32106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:53:54.972510 32106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:53:54.998258 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.986022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:43709 (local address 127.0.0.1:59534)
0619 16:53:54.988238 (+ 2216us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:54.988304 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:54.988537 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:54.990160 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:54.990170 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:54.990209 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:54.990645 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.990660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.992740 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.992745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.995310 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.995325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.997251 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.997257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.997444 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.997497 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 16:53:54.997641 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:53:54.997846 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1878,"thread_start_us":1695,"threads_started":1}
I20260619 16:53:54.998767 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.986248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.14.153.2:43709)
0619 16:53:54.987335 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:54.987347 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:54.988800 (+ 1453us) server_negotiation.cc:407] Connection header received
0619 16:53:54.989582 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:54.989593 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:54.989897 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:54.990100 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:54.990875 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.992577 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.995471 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.996711 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.996862 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.997080 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:54.997961 (+ 881us) server_negotiation.cc:299] Negotiation successful
0619 16:53:54.998158 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":1110,"threads_started":1}
I20260619 16:53:54.998944 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.989005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:43133 (local address 127.0.0.1:46696)
0619 16:53:54.991510 (+ 2505us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:54.991525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:54.991661 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:54.992159 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:54.992163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:54.992188 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:54.992543 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.992552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.994228 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.994234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.996018 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.996033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.998092 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.998100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.998290 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.998325 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:53:54.998406 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:53:54.998518 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2384,"mutex_wait_us":1,"thread_start_us":2265,"threads_started":1}
I20260619 16:53:54.999502 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.987091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.14.153.3:36253)
0619 16:53:54.989648 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:54.989660 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:54.989699 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:53:54.989812 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:54.989822 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:54.990087 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:54.990299 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:54.991111 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.993420 (+ 2309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.995854 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.997246 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.997374 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.997602 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:54.998203 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 16:53:54.998534 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2379,"thread_start_us":2224,"threads_started":1}
I20260619 16:53:54.999445 32579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.986929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:36253 (local address 127.0.0.1:43504)
0619 16:53:54.988928 (+ 1999us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:54.988944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:54.989049 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:54.990384 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:54.990388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:54.990408 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:54.990853 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.990864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.993583 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.993590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.995707 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:54.995718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.997752 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.997758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:54.997961 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.998006 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 16:53:54.999005 (+ 999us) client_negotiation.cc:240] Negotiation successful
0619 16:53:54.999192 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1856,"mutex_wait_us":1,"thread_start_us":1678,"threads_started":1}
I20260619 16:53:54.999758 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:54.989499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.14.153.1:43133)
0619 16:53:54.991062 (+ 1563us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:54.991074 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:54.991636 (+ 562us) server_negotiation.cc:407] Connection header received
0619 16:53:54.991810 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:54.991815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:54.991983 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:54.992167 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:54.992736 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.994084 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.996181 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:54.997570 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:54.997731 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:54.997947 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:54.999210 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0619 16:53:54.999368 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":1227,"threads_started":1}
I20260619 16:53:55.002980 32365 tablet_service.cc:1467] Processing CreateTablet for tablet a3b2d829c73645f1b3bff82c45411e20 (DEFAULT_TABLE table=test-table [id=39660b849c234c9e812a6547cb195c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:55.003396 32231 tablet_service.cc:1467] Processing CreateTablet for tablet a3b2d829c73645f1b3bff82c45411e20 (DEFAULT_TABLE table=test-table [id=39660b849c234c9e812a6547cb195c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:55.003396 32499 tablet_service.cc:1467] Processing CreateTablet for tablet a3b2d829c73645f1b3bff82c45411e20 (DEFAULT_TABLE table=test-table [id=39660b849c234c9e812a6547cb195c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:55.004411 32365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b2d829c73645f1b3bff82c45411e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:55.004801 32231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b2d829c73645f1b3bff82c45411e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:55.004817 32499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b2d829c73645f1b3bff82c45411e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:55.017539 32583 tablet_bootstrap.cc:492] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: Bootstrap starting.
I20260619 16:53:55.019702 32584 tablet_bootstrap.cc:492] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: Bootstrap starting.
I20260619 16:53:55.020577 32585 tablet_bootstrap.cc:492] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: Bootstrap starting.
I20260619 16:53:55.022063 32583 tablet_bootstrap.cc:654] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:55.023499 32583 log.cc:824] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:55.024380 32584 tablet_bootstrap.cc:654] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:55.024909 32585 tablet_bootstrap.cc:654] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:55.025869 32584 log.cc:824] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:55.026175 32585 log.cc:824] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:55.027807 32583 tablet_bootstrap.cc:492] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: No bootstrap required, opened a new log
I20260619 16:53:55.028095 32583 ts_tablet_manager.cc:1397] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 16:53:55.029240 32585 tablet_bootstrap.cc:492] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: No bootstrap required, opened a new log
I20260619 16:53:55.029240 32584 tablet_bootstrap.cc:492] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: No bootstrap required, opened a new log
I20260619 16:53:55.029510 32584 ts_tablet_manager.cc:1397] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260619 16:53:55.029510 32585 ts_tablet_manager.cc:1397] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 16:53:55.035559 32583 raft_consensus.cc:348] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } }
I20260619 16:53:55.035959 32583 raft_consensus.cc:374] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:55.036105 32583 raft_consensus.cc:729] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e1df91dd5f422f9445ec34a930689e, State: Initialized, Role: FOLLOWER
I20260619 16:53:55.036948 32583 consensus_queue.cc:260] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } }
I20260619 16:53:55.038090 32430 heartbeater.cc:503] Master 127.14.153.62:41153 was elected leader, sending a full tablet report...
I20260619 16:53:55.038791 32583 ts_tablet_manager.cc:1428] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 16:53:55.039882 32585 raft_consensus.cc:348] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } }
I20260619 16:53:55.040256 32585 raft_consensus.cc:374] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:55.040381 32585 raft_consensus.cc:729] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9613bdc21e8540c7af2e17ece248a98f, State: Initialized, Role: FOLLOWER
I20260619 16:53:55.040977 32585 consensus_queue.cc:260] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } }
I20260619 16:53:55.041097 32584 raft_consensus.cc:348] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } }
I20260619 16:53:55.041604 32584 raft_consensus.cc:374] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:55.041750 32584 raft_consensus.cc:729] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce04b3fbe0ed476e9db8c1ff68c9ae01, State: Initialized, Role: FOLLOWER
I20260619 16:53:55.042100 32564 heartbeater.cc:503] Master 127.14.153.62:41153 was elected leader, sending a full tablet report...
I20260619 16:53:55.042508 32584 consensus_queue.cc:260] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } }
I20260619 16:53:55.042726 32585 ts_tablet_manager.cc:1428] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 16:53:55.044708 32584 ts_tablet_manager.cc:1428] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
W20260619 16:53:55.094707 32297 tablet.cc:2369] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:55.110313 32590 raft_consensus.cc:482] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:55.110715 32590 raft_consensus.cc:504] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } }
I20260619 16:53:55.112411 32590 leader_election.cc:288] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e1df91dd5f422f9445ec34a930689e (127.14.153.2:43709), ce04b3fbe0ed476e9db8c1ff68c9ae01 (127.14.153.1:43133)
W20260619 16:53:55.115764 32431 tablet.cc:2369] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:55.119143 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:55.112966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:40231 (local address 127.14.153.1:43133)
0619 16:53:55.113180 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:55.113186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:55.113206 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:53:55.113416 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:55.113421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:55.113508 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:55.113629 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:55.113987 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.114995 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.116985 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.118210 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.118346 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:55.118401 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:55.118747 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 16:53:55.118872 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 16:53:55.119163 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:55.112727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:43133 (local address 127.14.153.3:40231)
0619 16:53:55.113052 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:55.113074 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:55.113227 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:55.113665 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:55.113668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:55.113692 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:55.113848 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:55.113856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.115180 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.115186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:55.116831 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:55.116843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.118466 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.118470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:55.118596 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:55.118619 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:53:55.118721 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:53:55.118853 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260619 16:53:55.120165 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:55.112819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:43709 (local address 127.14.153.3:53719)
0619 16:53:55.113716 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:55.113736 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:55.113861 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:55.114221 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:55.114226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:55.114251 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:55.114639 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:55.114651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.116085 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.116090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:55.118117 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:55.118134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.119446 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.119452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:55.119648 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:55.119682 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:53:55.119773 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:53:55.119904 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":1,"thread_start_us":558,"threads_started":1}
I20260619 16:53:55.120230 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:55.112968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:53719 (local address 127.14.153.2:43709)
0619 16:53:55.113170 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:55.113177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:55.113812 (+ 635us) server_negotiation.cc:407] Connection header received
0619 16:53:55.114038 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:55.114043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:55.114101 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:55.114177 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:55.114800 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.115935 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.118290 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.119229 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.119399 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:55.119462 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:55.119818 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 16:53:55.119962 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:53:55.120430 32251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b2d829c73645f1b3bff82c45411e20" candidate_uuid: "9613bdc21e8540c7af2e17ece248a98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" is_pre_election: true
I20260619 16:53:55.121042 32251 raft_consensus.cc:2454] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9613bdc21e8540c7af2e17ece248a98f in term 0.
I20260619 16:53:55.121300 32385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b2d829c73645f1b3bff82c45411e20" candidate_uuid: "9613bdc21e8540c7af2e17ece248a98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e1df91dd5f422f9445ec34a930689e" is_pre_election: true
I20260619 16:53:55.121726 32454 leader_election.cc:302] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9613bdc21e8540c7af2e17ece248a98f, ce04b3fbe0ed476e9db8c1ff68c9ae01; no voters:
I20260619 16:53:55.121881 32385 raft_consensus.cc:2454] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9613bdc21e8540c7af2e17ece248a98f in term 0.
I20260619 16:53:55.122107 32590 raft_consensus.cc:2781] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:53:55.122237 32590 raft_consensus.cc:482] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:53:55.122363 32590 raft_consensus.cc:3037] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:55.124647 32590 raft_consensus.cc:504] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } }
I20260619 16:53:55.125389 32590 leader_election.cc:288] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [CANDIDATE]: Term 1 election: Requested vote from peers 43e1df91dd5f422f9445ec34a930689e (127.14.153.2:43709), ce04b3fbe0ed476e9db8c1ff68c9ae01 (127.14.153.1:43133)
I20260619 16:53:55.125749 32385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b2d829c73645f1b3bff82c45411e20" candidate_uuid: "9613bdc21e8540c7af2e17ece248a98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e1df91dd5f422f9445ec34a930689e"
I20260619 16:53:55.125867 32251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b2d829c73645f1b3bff82c45411e20" candidate_uuid: "9613bdc21e8540c7af2e17ece248a98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01"
I20260619 16:53:55.125955 32385 raft_consensus.cc:3037] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:55.126106 32251 raft_consensus.cc:3037] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:55.128245 32385 raft_consensus.cc:2454] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9613bdc21e8540c7af2e17ece248a98f in term 1.
I20260619 16:53:55.128357 32251 raft_consensus.cc:2454] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9613bdc21e8540c7af2e17ece248a98f in term 1.
I20260619 16:53:55.128674 32453 leader_election.cc:302] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e1df91dd5f422f9445ec34a930689e, 9613bdc21e8540c7af2e17ece248a98f; no voters:
I20260619 16:53:55.128993 32590 raft_consensus.cc:2781] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:55.129454 32590 raft_consensus.cc:686] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 1 LEADER]: Becoming Leader. State: Replica: 9613bdc21e8540c7af2e17ece248a98f, State: Running, Role: LEADER
I20260619 16:53:55.129944 32590 consensus_queue.cc:237] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } }
I20260619 16:53:55.135252 32105 catalog_manager.cc:5495] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9613bdc21e8540c7af2e17ece248a98f (127.14.153.3). New cstate: current_term: 1 leader_uuid: "9613bdc21e8540c7af2e17ece248a98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9613bdc21e8540c7af2e17ece248a98f" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 } health_report { overall_health: UNKNOWN } } }
I20260619 16:53:55.151450 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:55.146064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.14.153.3:36253)
0619 16:53:55.146260 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:55.146265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:55.146300 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:53:55.146476 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:55.146480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:55.146569 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:55.146694 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:55.147010 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.147809 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.149251 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.149743 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.149833 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:55.150341 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:55.150885 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:55.151004 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:55.151176 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:53:55.151242 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:53:55.159871 32385 raft_consensus.cc:1261] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [term 1 FOLLOWER]: Refusing update from remote peer 9613bdc21e8540c7af2e17ece248a98f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:55.159871 32251 raft_consensus.cc:1261] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [term 1 FOLLOWER]: Refusing update from remote peer 9613bdc21e8540c7af2e17ece248a98f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 16:53:55.159873 32565 tablet.cc:2369] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:55.160957 32593 consensus_queue.cc:1035] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e1df91dd5f422f9445ec34a930689e" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 43709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:55.161334 32590 consensus_queue.cc:1035] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce04b3fbe0ed476e9db8c1ff68c9ae01" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:55.186561 32105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:55.186872 32105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:55.190889 32105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71917f3d1911443db1aad9ae077718ae: Sending DeleteTablet for 3 replicas of tablet a3b2d829c73645f1b3bff82c45411e20
I20260619 16:53:55.192167 32365 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b2d829c73645f1b3bff82c45411e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:55 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:59534
I20260619 16:53:55.192165 32499 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b2d829c73645f1b3bff82c45411e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:55 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:43504
I20260619 16:53:55.192346 32231 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b2d829c73645f1b3bff82c45411e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:55 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:46696
I20260619 16:53:55.194124 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 32168
I20260619 16:53:55.205765 32606 tablet_replica.cc:330] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: stopping tablet replica
I20260619 16:53:55.206403 32606 raft_consensus.cc:2229] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:53:55.207273 32606 raft_consensus.cc:2258] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:55.208007 32607 tablet_replica.cc:330] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: stopping tablet replica
I20260619 16:53:55.208534 32607 raft_consensus.cc:2229] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:55.209225 32607 raft_consensus.cc:2258] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:55.210100 32605 tablet_replica.cc:330] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: stopping tablet replica
I20260619 16:53:55.210530 32605 raft_consensus.cc:2229] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:55.211120 32605 raft_consensus.cc:2258] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:55.211784 32607 ts_tablet_manager.cc:1905] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:55.213315 32605 ts_tablet_manager.cc:1905] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:55.219234 32607 ts_tablet_manager.cc:1918] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:55.219458 32607 log.cc:1196] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/a3b2d829c73645f1b3bff82c45411e20
I20260619 16:53:55.220039 32607 ts_tablet_manager.cc:1939] T a3b2d829c73645f1b3bff82c45411e20 P ce04b3fbe0ed476e9db8c1ff68c9ae01: Deleting consensus metadata
I20260619 16:53:55.220505 32605 ts_tablet_manager.cc:1918] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:55.220724 32605 log.cc:1196] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/a3b2d829c73645f1b3bff82c45411e20
I20260619 16:53:55.221302 32605 ts_tablet_manager.cc:1939] T a3b2d829c73645f1b3bff82c45411e20 P 43e1df91dd5f422f9445ec34a930689e: Deleting consensus metadata
I20260619 16:53:55.222210 32093 catalog_manager.cc:4841] TS ce04b3fbe0ed476e9db8c1ff68c9ae01 (127.14.153.1:43133): tablet a3b2d829c73645f1b3bff82c45411e20 (table test-table [id=39660b849c234c9e812a6547cb195c5c]) successfully deleted
I20260619 16:53:55.223043 32092 catalog_manager.cc:4841] TS 43e1df91dd5f422f9445ec34a930689e (127.14.153.2:43709): tablet a3b2d829c73645f1b3bff82c45411e20 (table test-table [id=39660b849c234c9e812a6547cb195c5c]) successfully deleted
I20260619 16:53:55.224524 32606 ts_tablet_manager.cc:1905] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:55.231120 32606 ts_tablet_manager.cc:1918] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 19 16:53:55 dist-test-slave-qb36 krb5kdc[32062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:53:55.231349 32606 log.cc:1196] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/a3b2d829c73645f1b3bff82c45411e20
I20260619 16:53:55.231974 32606 ts_tablet_manager.cc:1939] T a3b2d829c73645f1b3bff82c45411e20 P 9613bdc21e8540c7af2e17ece248a98f: Deleting consensus metadata
I20260619 16:53:55.233879 32093 catalog_manager.cc:4841] TS 9613bdc21e8540c7af2e17ece248a98f (127.14.153.3:36253): tablet a3b2d829c73645f1b3bff82c45411e20 (table test-table [id=39660b849c234c9e812a6547cb195c5c]) successfully deleted
I20260619 16:53:55.239192 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:55.196047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.14.153.1:43133)
0619 16:53:55.196236 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:55.196242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:55.203594 (+ 7352us) server_negotiation.cc:407] Connection header received
0619 16:53:55.208033 (+ 4439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:55.208040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:55.208134 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:55.208262 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:55.217610 (+ 9348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.218795 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.223405 (+ 4610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.224066 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.224175 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:55.232762 (+ 8587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:55.232781 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:55.232800 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:55.232846 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:55.236287 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:55.236994 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:55.237002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:55.237004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:55.237135 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:55.237527 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:55.237531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:55.237533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:55.237899 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:55.238216 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:55.238512 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 16:53:55.238748 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:53:55.239910 32291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:55.381635 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 32168
I20260619 16:53:55.399724 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 32302
Jun 19 16:53:55 dist-test-slave-qb36 krb5kdc[32062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:53:55.416937 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:55.400613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.14.153.2:43709)
0619 16:53:55.400834 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:55.400841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:55.400864 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:53:55.401028 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:55.401034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:55.401118 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:55.401235 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:55.402447 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.403618 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.405095 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.405856 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.405973 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:55.410119 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:55.410138 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:55.410160 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:55.410212 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:55.413968 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:55.414641 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:55.414651 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:55.414655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:55.414814 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:55.415231 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:55.415236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:55.415240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:55.415750 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:55.416075 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:55.416319 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:53:55.416543 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:53:55.417688 32425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:55.509130 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 32302
I20260619 16:53:55.532315 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 32436
Jun 19 16:53:55 dist-test-slave-qb36 krb5kdc[32062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:53:55.552354 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:55.533102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.14.153.3:36253)
0619 16:53:55.533310 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:55.533318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:55.533559 (+ 241us) server_negotiation.cc:407] Connection header received
0619 16:53:55.533724 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:55.533728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:55.533826 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:55.533917 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:55.535870 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.536726 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.538635 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:55.539344 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:55.539450 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:55.544878 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:55.544894 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:55.544914 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:55.544970 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:55.549157 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:55.550122 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:55.550132 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:55.550135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:55.550348 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:55.550807 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:55.550812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:55.550815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:55.551262 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:55.551491 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:55.551835 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:53:55.552005 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:53:55.553066 32559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:55.677542 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 32436
I20260619 16:53:55.697849 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 32075
I20260619 16:53:55.699005 32137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:55.812387 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 32075
2026-06-19T16:53:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4437 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T16:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:55Z Disabled control of system clock
I20260619 16:53:55.888197 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35753
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:56.222018 32616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:56.222318 32616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:56.222452 32616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:56.229907 32616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:56.230012 32616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:56.230091 32616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:56.230192 32616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:56.230266 32616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:56.239857 32616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:56.241298 32616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:56.243160 32616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:56.255470 32621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:56.255437 32622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:56.255537 32624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:56.256177 32616 server_base.cc:1034] running on GCE node
I20260619 16:53:56.256965 32616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:56.258383 32616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:56.259610 32616 hybrid_clock.cc:648] HybridClock initialized: now 1781888036259566 us; error 36 us; skew 500 ppm
I20260619 16:53:56.260069 32616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:53:56.265232 32616 webserver.cc:460] Webserver started at http://127.14.153.62:40233/ using document root <none> and password file <none>
I20260619 16:53:56.265988 32616 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:56.266120 32616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:56.266472 32616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:56.269526 32616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "438d20016e2f4d37834bffb4656dc1af"
format_stamp: "Formatted at 2026-06-19 16:53:56 on dist-test-slave-qb36"
server_key: "c23daf7f701169dd2fa96926fb085a30"
server_key_iv: "5b9e749a99ba0fd0b8d5329b54fb3875"
server_key_version: "encryptionkey@0"
I20260619 16:53:56.270380 32616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "438d20016e2f4d37834bffb4656dc1af"
format_stamp: "Formatted at 2026-06-19 16:53:56 on dist-test-slave-qb36"
server_key: "c23daf7f701169dd2fa96926fb085a30"
server_key_iv: "5b9e749a99ba0fd0b8d5329b54fb3875"
server_key_version: "encryptionkey@0"
I20260619 16:53:56.275770 32616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 16:53:56.279477 32630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:56.281102 32616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:53:56.281316 32616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "438d20016e2f4d37834bffb4656dc1af"
format_stamp: "Formatted at 2026-06-19 16:53:56 on dist-test-slave-qb36"
server_key: "c23daf7f701169dd2fa96926fb085a30"
server_key_iv: "5b9e749a99ba0fd0b8d5329b54fb3875"
server_key_version: "encryptionkey@0"
I20260619 16:53:56.281592 32616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:56.293334 32616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:56.294147 32616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:56.294445 32616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:56.312618 32616 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35753
I20260619 16:53:56.312674 32681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35753 every 8 connection(s)
I20260619 16:53:56.314002 32616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:56.317859 32682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:56.321360 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 32616
I20260619 16:53:56.321542 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:53:56.322134 14948 external_mini_cluster.cc:1428] Setting key e81785555a3b43f70583430cd122701a
I20260619 16:53:56.330672 32682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af: Bootstrap starting.
I20260619 16:53:56.334333 32682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:56.335702 32682 log.cc:824] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:56.336670 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:56.324116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.14.153.62:35753)
0619 16:53:56.325109 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:56.325120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:56.325165 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:53:56.326776 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:56.326801 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:56.327379 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:56.327722 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:56.328342 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:56.329592 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:56.331442 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:56.332118 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:56.332257 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:56.333625 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:56.333632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:56.333652 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:56.334191 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:56.334286 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:56.334307 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:56.334398 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:56.334994 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 16:53:56.335341 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":366,"threads_started":1}
I20260619 16:53:56.338755 32682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af: No bootstrap required, opened a new log
I20260619 16:53:56.346092 32682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438d20016e2f4d37834bffb4656dc1af" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35753 } }
I20260619 16:53:56.346490 32682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:56.346604 32682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 438d20016e2f4d37834bffb4656dc1af, State: Initialized, Role: FOLLOWER
I20260619 16:53:56.347227 32682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438d20016e2f4d37834bffb4656dc1af" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35753 } }
I20260619 16:53:56.347465 32682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:56.347618 32682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:56.347786 32682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:56.349975 32682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438d20016e2f4d37834bffb4656dc1af" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35753 } }
I20260619 16:53:56.350512 32682 leader_election.cc:302] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 438d20016e2f4d37834bffb4656dc1af; no voters:
I20260619 16:53:56.351147 32682 leader_election.cc:288] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:56.351307 32687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:56.352010 32687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [term 1 LEADER]: Becoming Leader. State: Replica: 438d20016e2f4d37834bffb4656dc1af, State: Running, Role: LEADER
I20260619 16:53:56.352573 32687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438d20016e2f4d37834bffb4656dc1af" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35753 } }
I20260619 16:53:56.353226 32682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:56.355474 32688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "438d20016e2f4d37834bffb4656dc1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438d20016e2f4d37834bffb4656dc1af" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35753 } } }
I20260619 16:53:56.355759 32689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 438d20016e2f4d37834bffb4656dc1af. Latest consensus state: current_term: 1 leader_uuid: "438d20016e2f4d37834bffb4656dc1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438d20016e2f4d37834bffb4656dc1af" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35753 } } }
I20260619 16:53:56.355967 32688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:56.356081 32689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:56.357581 32695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:56.362979 32695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:56.371161 32695 catalog_manager.cc:1347] Generated new cluster ID: 7429997b55384fa4b2df071e1ac2062c
I20260619 16:53:56.371277 32695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:56.396482 32695 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:56.398461 32695 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:56.410332 32695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af: Generated new TSK 0
I20260619 16:53:56.411463 32695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:56.424504 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35753
--builtin_ntp_servers=127.14.153.20:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:56.759752 32706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:56.760082 32706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:56.760255 32706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:56.767622 32706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:56.767783 32706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:56.767961 32706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:53:56.777184 32706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:56.778532 32706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:56.780440 32706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:56.794111 32711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:56.794132 32714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:56.794068 32712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:56.794927 32706 server_base.cc:1034] running on GCE node
I20260619 16:53:56.795558 32706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:56.796445 32706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:56.797678 32706 hybrid_clock.cc:648] HybridClock initialized: now 1781888036797586 us; error 63 us; skew 500 ppm
I20260619 16:53:56.798035 32706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:53:56.800351 32706 webserver.cc:460] Webserver started at http://127.14.153.1:41969/ using document root <none> and password file <none>
I20260619 16:53:56.801003 32706 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:56.801088 32706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:56.801352 32706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:56.804234 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "80b04f35fe0c47318f84bf429e6aa0a3"
format_stamp: "Formatted at 2026-06-19 16:53:56 on dist-test-slave-qb36"
server_key: "65d3561b1763c7bf84662995696e14a2"
server_key_iv: "a905cf70759bede3abe3a97c332472f6"
server_key_version: "encryptionkey@0"
I20260619 16:53:56.804966 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "80b04f35fe0c47318f84bf429e6aa0a3"
format_stamp: "Formatted at 2026-06-19 16:53:56 on dist-test-slave-qb36"
server_key: "65d3561b1763c7bf84662995696e14a2"
server_key_iv: "a905cf70759bede3abe3a97c332472f6"
server_key_version: "encryptionkey@0"
I20260619 16:53:56.810130 32706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:53:56.813596 32720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:56.814931 32706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:53:56.815121 32706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "80b04f35fe0c47318f84bf429e6aa0a3"
format_stamp: "Formatted at 2026-06-19 16:53:56 on dist-test-slave-qb36"
server_key: "65d3561b1763c7bf84662995696e14a2"
server_key_iv: "a905cf70759bede3abe3a97c332472f6"
server_key_version: "encryptionkey@0"
I20260619 16:53:56.815346 32706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:56.825762 32706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:56.826542 32706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:56.826841 32706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:56.827862 32706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:56.829303 32706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:56.829389 32706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:56.829536 32706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:56.829640 32706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:56.862746 32706 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:41111
I20260619 16:53:56.862785 364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:41111 every 8 connection(s)
I20260619 16:53:56.864226 32706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:53:56.871403 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 32706
I20260619 16:53:56.871588 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:53:56.875731 14948 external_mini_cluster.cc:1428] Setting key 4ff97c313d49ed95ae4c03bf43443e88
I20260619 16:53:56.876669 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:56.866718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:59227 (local address 127.14.153.62:35753)
0619 16:53:56.867208 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:56.867214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:56.868372 (+ 1158us) server_negotiation.cc:407] Connection header received
0619 16:53:56.870170 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:56.870175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:56.870262 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:56.870376 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:56.871476 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:56.872279 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:56.873813 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:56.874220 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:56.874346 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:56.875769 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:56.875775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:56.875780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:56.875804 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:56.875868 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:56.875885 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:56.875972 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:56.876190 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 16:53:56.876292 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":221,"threads_started":1}
I20260619 16:53:56.877746 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:56.867360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35753 (local address 127.14.153.1:59227)
0619 16:53:56.868224 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:56.868260 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:56.869897 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:56.870608 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:56.870616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:56.870709 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:56.871315 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:56.871332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:56.872434 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:56.872438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:56.873666 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:56.873678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:56.874442 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:56.874449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:56.874680 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:56.875435 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:56.875438 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:56.875482 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:56.875571 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:56.875969 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:56.875981 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:56.875990 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:53:56.876181 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 16:53:56.876584 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":251,"threads_started":1}
I20260619 16:53:56.878470 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35753
--builtin_ntp_servers=127.14.153.20:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:53:56.879432 365 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35753
I20260619 16:53:56.879746 365 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:56.880457 365 heartbeater.cc:511] Master 127.14.153.62:35753 requested a full tablet report, sending...
I20260619 16:53:56.882827 32647 ts_manager.cc:194] Registered new tserver with Master: 80b04f35fe0c47318f84bf429e6aa0a3 (127.14.153.1:41111)
I20260619 16:53:56.885088 32647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.1:59227
W20260619 16:53:57.210064 369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:57.210418 369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:57.210587 369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:57.217978 369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:57.218127 369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:57.218299 369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:53:57.227695 369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:57.229035 369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:57.230792 369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:57.243885 374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:57.243916 375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:57.244144 377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:57.244622 369 server_base.cc:1034] running on GCE node
I20260619 16:53:57.245246 369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:57.246236 369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:57.247437 369 hybrid_clock.cc:648] HybridClock initialized: now 1781888037247387 us; error 35 us; skew 500 ppm
I20260619 16:53:57.247802 369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:53:57.250065 369 webserver.cc:460] Webserver started at http://127.14.153.2:46493/ using document root <none> and password file <none>
I20260619 16:53:57.250761 369 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:57.250869 369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:57.251269 369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:57.254112 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "360bfadc4b304c71901dc439fa400b34"
format_stamp: "Formatted at 2026-06-19 16:53:57 on dist-test-slave-qb36"
server_key: "277db9b4f07497568f3c5676dcd93005"
server_key_iv: "76fd79170c1866f675e65db5122273a1"
server_key_version: "encryptionkey@0"
I20260619 16:53:57.254928 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "360bfadc4b304c71901dc439fa400b34"
format_stamp: "Formatted at 2026-06-19 16:53:57 on dist-test-slave-qb36"
server_key: "277db9b4f07497568f3c5676dcd93005"
server_key_iv: "76fd79170c1866f675e65db5122273a1"
server_key_version: "encryptionkey@0"
I20260619 16:53:57.260188 369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 16:53:57.263743 383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:57.264987 369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:53:57.265164 369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "360bfadc4b304c71901dc439fa400b34"
format_stamp: "Formatted at 2026-06-19 16:53:57 on dist-test-slave-qb36"
server_key: "277db9b4f07497568f3c5676dcd93005"
server_key_iv: "76fd79170c1866f675e65db5122273a1"
server_key_version: "encryptionkey@0"
I20260619 16:53:57.265383 369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:57.282672 369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:57.283610 369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:57.283917 369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:57.284948 369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:57.286481 369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:57.286609 369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:57.286731 369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:57.286816 369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:57.320112 369 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:39349
I20260619 16:53:57.320153 495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:39349 every 8 connection(s)
I20260619 16:53:57.321491 369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:57.325598 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 369
I20260619 16:53:57.325778 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:53:57.326368 14948 external_mini_cluster.cc:1428] Setting key 0d57939eda5ebd7ca5167c5cf6f31a2f
I20260619 16:53:57.329560 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35753
--builtin_ntp_servers=127.14.153.20:40545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:53:57.336045 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.324307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:56361 (local address 127.14.153.62:35753)
0619 16:53:57.324492 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:57.324498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:57.326079 (+ 1581us) server_negotiation.cc:407] Connection header received
0619 16:53:57.328233 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:57.328238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:57.328324 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:57.328437 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:57.329742 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.330888 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.332829 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.333536 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.333692 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.335064 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:57.335073 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:57.335081 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:57.335108 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:57.335151 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:57.335170 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:57.335267 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:57.335520 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 16:53:57.335653 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:53:57.337270 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.324904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35753 (local address 127.14.153.2:56361)
0619 16:53:57.325928 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:57.325966 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:57.327976 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:57.328689 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:57.328701 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:57.328817 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:57.329552 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.329575 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.331122 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.331128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.332677 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.332690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.333761 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.333766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.333968 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.334722 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:57.334724 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:57.334766 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:57.334849 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:57.335264 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:57.335272 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:57.335278 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:53:57.335467 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 16:53:57.335912 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":1868,"threads_started":1}
I20260619 16:53:57.338738 496 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35753
I20260619 16:53:57.339076 496 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:57.339851 496 heartbeater.cc:511] Master 127.14.153.62:35753 requested a full tablet report, sending...
I20260619 16:53:57.341297 32647 ts_manager.cc:194] Registered new tserver with Master: 360bfadc4b304c71901dc439fa400b34 (127.14.153.2:39349)
I20260619 16:53:57.342084 32647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.2:56361
W20260619 16:53:57.656589 499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:57.656920 499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:57.657073 499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:57.664400 499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:57.664525 499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:57.664710 499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:53:57.673782 499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:35753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:53:57.675163 499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:57.676986 499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:57.690666 504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:57.690696 507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:57.690834 505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:57.691576 499 server_base.cc:1034] running on GCE node
I20260619 16:53:57.692224 499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:57.693238 499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:57.694468 499 hybrid_clock.cc:648] HybridClock initialized: now 1781888037694422 us; error 38 us; skew 500 ppm
I20260619 16:53:57.694797 499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:53:57.697129 499 webserver.cc:460] Webserver started at http://127.14.153.3:46739/ using document root <none> and password file <none>
I20260619 16:53:57.697844 499 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:57.697973 499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:57.698282 499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:57.701228 499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "e7b49dfdde704845b3614c4c341b05e2"
format_stamp: "Formatted at 2026-06-19 16:53:57 on dist-test-slave-qb36"
server_key: "100eb46010c1158015e3cb90ff85f821"
server_key_iv: "6b87a34c443313f65c5b71adefaf1ef1"
server_key_version: "encryptionkey@0"
I20260619 16:53:57.702028 499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "e7b49dfdde704845b3614c4c341b05e2"
format_stamp: "Formatted at 2026-06-19 16:53:57 on dist-test-slave-qb36"
server_key: "100eb46010c1158015e3cb90ff85f821"
server_key_iv: "6b87a34c443313f65c5b71adefaf1ef1"
server_key_version: "encryptionkey@0"
I20260619 16:53:57.707191 499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:53:57.710662 513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:57.712028 499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:57.712203 499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "e7b49dfdde704845b3614c4c341b05e2"
format_stamp: "Formatted at 2026-06-19 16:53:57 on dist-test-slave-qb36"
server_key: "100eb46010c1158015e3cb90ff85f821"
server_key_iv: "6b87a34c443313f65c5b71adefaf1ef1"
server_key_version: "encryptionkey@0"
I20260619 16:53:57.712427 499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:57.723223 499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:57.724043 499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:57.724346 499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:57.725303 499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:57.726693 499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:57.726809 499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:57.726938 499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:57.727025 499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:57.759945 499 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:42311
I20260619 16:53:57.760033 625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:42311 every 8 connection(s)
I20260619 16:53:57.761329 499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:57.764858 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 499
I20260619 16:53:57.765056 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:53:57.765609 14948 external_mini_cluster.cc:1428] Setting key 3a249e4a3aeb3faa3fc9e1bad5afd20b
I20260619 16:53:57.773792 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.763918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:50653 (local address 127.14.153.62:35753)
0619 16:53:57.764096 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:57.764101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:57.765736 (+ 1635us) server_negotiation.cc:407] Connection header received
0619 16:53:57.767917 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:57.767922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:57.767983 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:57.768062 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:57.769171 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.769947 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.771481 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.771890 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.771981 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.773166 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:57.773170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:57.773173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:57.773191 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:57.773221 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:57.773233 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:57.773301 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:57.773479 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:53:57.773551 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:53:57.774739 628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.764481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:35753 (local address 127.14.153.3:50653)
0619 16:53:57.765550 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:57.765601 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:57.767635 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:57.768311 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:57.768319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:57.768437 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:57.769010 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.769027 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.770097 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.770101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.771369 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.771382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.772062 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.772066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.772256 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.772878 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:57.772881 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:57.772919 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:57.772996 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:57.773310 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:57.773317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:57.773322 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:53:57.773457 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 16:53:57.773739 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":281,"threads_started":1}
I20260619 16:53:57.776079 626 heartbeater.cc:347] Connected to a master server at 127.14.153.62:35753
I20260619 16:53:57.776341 626 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:57.777019 626 heartbeater.cc:511] Master 127.14.153.62:35753 requested a full tablet report, sending...
I20260619 16:53:57.778354 32647 ts_manager.cc:194] Registered new tserver with Master: e7b49dfdde704845b3614c4c341b05e2 (127.14.153.3:42311)
I20260619 16:53:57.779277 32647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.3:50653
I20260619 16:53:57.781203 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:53:57.793747 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.786416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.14.153.62:35753)
0619 16:53:57.786674 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:57.786703 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:57.787463 (+ 760us) server_negotiation.cc:407] Connection header received
0619 16:53:57.787776 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:57.787784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:57.787882 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:57.787959 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:57.788793 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.789916 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.791398 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.791797 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.791933 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.793019 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:57.793030 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:57.793033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:57.793051 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:57.793082 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:57.793094 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:57.793191 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:57.793337 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 16:53:57.793441 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:53:57.797474 32647 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:53:57.804389 32647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:53:57.837852 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.817187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.14.153.2:39349)
0619 16:53:57.828155 (+ 10968us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:57.828165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:57.828196 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:53:57.828293 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:57.828302 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:57.828591 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:57.828804 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:57.829884 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.831635 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.834997 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.836640 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.836777 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.837067 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:57.837295 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 16:53:57.837467 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10769,"thread_start_us":10917,"threads_started":1}
I20260619 16:53:57.837841 640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.817006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:39349 (local address 127.0.0.1:47458)
0619 16:53:57.827704 (+ 10698us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:57.827746 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:57.827954 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:57.829148 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:57.829156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:57.829186 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:57.829682 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.829694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.831842 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.831850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.834813 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.834824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.836841 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.836847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.837047 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.837102 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 16:53:57.837231 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 16:53:57.837405 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9942,"mutex_wait_us":1,"thread_start_us":11942,"threads_started":1}
I20260619 16:53:57.839427 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.817446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:42311 (local address 127.0.0.1:35804)
0619 16:53:57.829245 (+ 11799us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:57.829267 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:57.829408 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:57.830368 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:57.830373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:57.830398 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:57.830688 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.830699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.833249 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.833257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.836070 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.836104 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.838577 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.838583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.838776 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.838808 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:53:57.838886 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:53:57.839047 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11648,"mutex_wait_us":1,"spinlock_wait_cycles":5888,"thread_start_us":9324,"threads_started":1}
I20260619 16:53:57.839718 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.818257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.14.153.3:42311)
0619 16:53:57.828803 (+ 10546us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:57.828811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:57.829536 (+ 725us) server_negotiation.cc:407] Connection header received
0619 16:53:57.829647 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:57.829656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:57.829900 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:57.830094 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:57.830891 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.833030 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.836284 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.838365 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.838811 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.839109 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:57.839190 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 16:53:57.839360 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10362,"thread_start_us":10203,"threads_started":1}
I20260619 16:53:57.841979 431 tablet_service.cc:1467] Processing CreateTablet for tablet 90f0a856bac0479180ee17ef2404e249 (DEFAULT_TABLE table=test-table [id=6b244afe08f14557bf62c64b0f6141f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:57.842605 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.830947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41111 (local address 127.0.0.1:34548)
0619 16:53:57.834603 (+ 3656us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:57.834621 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:57.834756 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:57.835515 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:57.835520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:57.835544 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:57.835957 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.835969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.837674 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.837679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.840076 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:57.840090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.841905 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.841912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:57.842089 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.842120 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:53:57.842210 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:53:57.842329 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3529,"thread_start_us":3300,"threads_started":1}
I20260619 16:53:57.842947 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:57.829555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.14.153.1:41111)
0619 16:53:57.834210 (+ 4655us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:57.834219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:57.834886 (+ 667us) server_negotiation.cc:407] Connection header received
0619 16:53:57.834985 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:57.834992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:57.835267 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:57.835488 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:57.836168 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.837528 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.840320 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:57.841703 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:57.841841 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:57.842085 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:57.842460 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 16:53:57.842622 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4482,"thread_start_us":4386,"threads_started":1}
I20260619 16:53:57.843672 431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f0a856bac0479180ee17ef2404e249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:57.843915 561 tablet_service.cc:1467] Processing CreateTablet for tablet 90f0a856bac0479180ee17ef2404e249 (DEFAULT_TABLE table=test-table [id=6b244afe08f14557bf62c64b0f6141f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:57.845109 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f0a856bac0479180ee17ef2404e249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:57.846387 300 tablet_service.cc:1467] Processing CreateTablet for tablet 90f0a856bac0479180ee17ef2404e249 (DEFAULT_TABLE table=test-table [id=6b244afe08f14557bf62c64b0f6141f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:57.847784 300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f0a856bac0479180ee17ef2404e249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:57.856953 644 tablet_bootstrap.cc:492] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: Bootstrap starting.
I20260619 16:53:57.860131 644 tablet_bootstrap.cc:654] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:57.861127 644 log.cc:824] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:57.863279 645 tablet_bootstrap.cc:492] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: Bootstrap starting.
I20260619 16:53:57.864156 644 tablet_bootstrap.cc:492] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: No bootstrap required, opened a new log
I20260619 16:53:57.864441 644 ts_tablet_manager.cc:1397] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:53:57.867161 645 tablet_bootstrap.cc:654] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:57.870067 645 log.cc:824] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:57.876665 644 raft_consensus.cc:348] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } }
I20260619 16:53:57.877291 644 raft_consensus.cc:374] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:57.877384 644 raft_consensus.cc:729] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b49dfdde704845b3614c4c341b05e2, State: Initialized, Role: FOLLOWER
I20260619 16:53:57.878365 644 consensus_queue.cc:260] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } }
I20260619 16:53:57.881698 647 tablet_bootstrap.cc:492] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3: Bootstrap starting.
I20260619 16:53:57.883718 645 tablet_bootstrap.cc:492] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: No bootstrap required, opened a new log
I20260619 16:53:57.883966 626 heartbeater.cc:503] Master 127.14.153.62:35753 was elected leader, sending a full tablet report...
I20260619 16:53:57.884155 645 ts_tablet_manager.cc:1397] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.007s
I20260619 16:53:57.884712 644 ts_tablet_manager.cc:1428] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260619 16:53:57.885802 647 tablet_bootstrap.cc:654] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:57.887197 647 log.cc:824] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:57.888788 365 heartbeater.cc:503] Master 127.14.153.62:35753 was elected leader, sending a full tablet report...
I20260619 16:53:57.891405 647 tablet_bootstrap.cc:492] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3: No bootstrap required, opened a new log
I20260619 16:53:57.891703 647 ts_tablet_manager.cc:1397] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 16:53:57.894512 645 raft_consensus.cc:348] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } }
I20260619 16:53:57.895006 645 raft_consensus.cc:374] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:57.895130 645 raft_consensus.cc:729] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360bfadc4b304c71901dc439fa400b34, State: Initialized, Role: FOLLOWER
I20260619 16:53:57.895855 645 consensus_queue.cc:260] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } }
I20260619 16:53:57.896962 496 heartbeater.cc:503] Master 127.14.153.62:35753 was elected leader, sending a full tablet report...
I20260619 16:53:57.897701 645 ts_tablet_manager.cc:1428] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260619 16:53:57.901082 647 raft_consensus.cc:348] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } }
I20260619 16:53:57.901541 647 raft_consensus.cc:374] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:57.901650 647 raft_consensus.cc:729] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80b04f35fe0c47318f84bf429e6aa0a3, State: Initialized, Role: FOLLOWER
I20260619 16:53:57.902196 647 consensus_queue.cc:260] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } }
I20260619 16:53:57.903777 647 ts_tablet_manager.cc:1428] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260619 16:53:58.013046 627 tablet.cc:2369] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:58.073624 497 tablet.cc:2369] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:58.116746 366 tablet.cc:2369] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:58.141323 652 raft_consensus.cc:482] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:58.141674 652 raft_consensus.cc:504] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } }
I20260619 16:53:58.143368 652 leader_election.cc:288] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360bfadc4b304c71901dc439fa400b34 (127.14.153.2:39349), e7b49dfdde704845b3614c4c341b05e2 (127.14.153.3:42311)
I20260619 16:53:58.152688 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.143933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:40349 (local address 127.14.153.3:42311)
0619 16:53:58.144129 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:58.144135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:58.144868 (+ 733us) server_negotiation.cc:407] Connection header received
0619 16:53:58.145079 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:58.145084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:58.145164 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:58.145269 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:58.146120 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.147256 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.150292 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.151557 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.151726 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:58.151798 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:58.152290 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 16:53:58.152419 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:53:58.152678 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.143974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:39859 (local address 127.14.153.2:39349)
0619 16:53:58.144173 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:58.144179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:58.144870 (+ 691us) server_negotiation.cc:407] Connection header received
0619 16:53:58.145079 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:58.145084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:58.145177 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:58.145269 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:58.146110 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.147256 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.150292 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.151557 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.151726 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:58.151798 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:58.152303 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 16:53:58.152419 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:53:58.152980 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.143688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:39349 (local address 127.14.153.1:39859)
0619 16:53:58.144705 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:58.144727 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:58.144943 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:58.145302 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:58.145307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:58.145342 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:58.145916 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:58.145935 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.147384 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.147388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:58.150044 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:58.150060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.151773 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.151781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:58.152098 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:58.152155 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 16:53:58.152252 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:53:58.152460 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":373,"spinlock_wait_cycles":43008,"thread_start_us":649,"threads_started":1}
I20260619 16:53:58.153231 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.143818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:42311 (local address 127.14.153.1:40349)
0619 16:53:58.144749 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:58.144767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:58.144906 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:58.145303 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:58.145307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:58.145349 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:58.145963 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:58.145974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.147385 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.147389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:58.150044 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:58.150060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.151791 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.151797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:58.152104 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:58.152162 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 16:53:58.152252 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:53:58.152460 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":322,"spinlock_wait_cycles":5760,"thread_start_us":509,"threads_started":1}
I20260619 16:53:58.154060 451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f0a856bac0479180ee17ef2404e249" candidate_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360bfadc4b304c71901dc439fa400b34" is_pre_election: true
I20260619 16:53:58.154261 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f0a856bac0479180ee17ef2404e249" candidate_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7b49dfdde704845b3614c4c341b05e2" is_pre_election: true
I20260619 16:53:58.154610 451 raft_consensus.cc:2454] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80b04f35fe0c47318f84bf429e6aa0a3 in term 0.
I20260619 16:53:58.154767 581 raft_consensus.cc:2454] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80b04f35fe0c47318f84bf429e6aa0a3 in term 0.
I20260619 16:53:58.155318 32721 leader_election.cc:302] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360bfadc4b304c71901dc439fa400b34, 80b04f35fe0c47318f84bf429e6aa0a3; no voters:
I20260619 16:53:58.155699 652 raft_consensus.cc:2781] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:53:58.155853 652 raft_consensus.cc:482] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:53:58.155953 652 raft_consensus.cc:3037] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:58.158140 652 raft_consensus.cc:504] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } }
I20260619 16:53:58.158793 652 leader_election.cc:288] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [CANDIDATE]: Term 1 election: Requested vote from peers 360bfadc4b304c71901dc439fa400b34 (127.14.153.2:39349), e7b49dfdde704845b3614c4c341b05e2 (127.14.153.3:42311)
I20260619 16:53:58.159224 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f0a856bac0479180ee17ef2404e249" candidate_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7b49dfdde704845b3614c4c341b05e2"
I20260619 16:53:58.159224 451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f0a856bac0479180ee17ef2404e249" candidate_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360bfadc4b304c71901dc439fa400b34"
I20260619 16:53:58.159420 451 raft_consensus.cc:3037] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:58.159413 581 raft_consensus.cc:3037] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:58.161559 451 raft_consensus.cc:2454] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80b04f35fe0c47318f84bf429e6aa0a3 in term 1.
I20260619 16:53:58.161597 581 raft_consensus.cc:2454] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80b04f35fe0c47318f84bf429e6aa0a3 in term 1.
I20260619 16:53:58.161988 32723 leader_election.cc:302] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 360bfadc4b304c71901dc439fa400b34, 80b04f35fe0c47318f84bf429e6aa0a3, e7b49dfdde704845b3614c4c341b05e2; no voters:
I20260619 16:53:58.162290 652 raft_consensus.cc:2781] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:58.162731 652 raft_consensus.cc:686] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 1 LEADER]: Becoming Leader. State: Replica: 80b04f35fe0c47318f84bf429e6aa0a3, State: Running, Role: LEADER
I20260619 16:53:58.163314 652 consensus_queue.cc:237] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } }
I20260619 16:53:58.167645 32647 catalog_manager.cc:5495] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80b04f35fe0c47318f84bf429e6aa0a3 (127.14.153.1). New cstate: current_term: 1 leader_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80b04f35fe0c47318f84bf429e6aa0a3" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 41111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 } health_report { overall_health: UNKNOWN } } }
I20260619 16:53:58.201015 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.196049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.14.153.1:41111)
0619 16:53:58.196230 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:58.196234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:58.196251 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:53:58.196338 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:58.196341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:58.196431 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:58.196551 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:58.196899 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.197600 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.198953 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.199367 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.199455 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:58.199919 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:58.200422 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:58.200535 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:58.200643 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 16:53:58.200720 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:53:58.209338 451 raft_consensus.cc:1261] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [term 1 FOLLOWER]: Refusing update from remote peer 80b04f35fe0c47318f84bf429e6aa0a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:58.209347 581 raft_consensus.cc:1261] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [term 1 FOLLOWER]: Refusing update from remote peer 80b04f35fe0c47318f84bf429e6aa0a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:58.210358 655 consensus_queue.cc:1035] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360bfadc4b304c71901dc439fa400b34" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:58.210754 652 consensus_queue.cc:1035] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7b49dfdde704845b3614c4c341b05e2" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 42311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:58.222282 658 mvcc.cc:205] Tried to move back new op lower bound from 7298613404494970880 to 7298613404320616448. Current Snapshot: MvccSnapshot[applied={T|T < 7298613404494970880}]
I20260619 16:53:58.223637 660 mvcc.cc:205] Tried to move back new op lower bound from 7298613404494970880 to 7298613404320616448. Current Snapshot: MvccSnapshot[applied={T|T < 7298613404494970880}]
I20260619 16:53:58.234866 32646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:58.235212 32646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:58.239316 32646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 438d20016e2f4d37834bffb4656dc1af: Sending DeleteTablet for 3 replicas of tablet 90f0a856bac0479180ee17ef2404e249
I20260619 16:53:58.240489 431 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f0a856bac0479180ee17ef2404e249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:58 UTC) from {username='slave'} at 127.0.0.1:47458
I20260619 16:53:58.240571 561 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f0a856bac0479180ee17ef2404e249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:58 UTC) from {username='slave'} at 127.0.0.1:35804
I20260619 16:53:58.240579 300 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f0a856bac0479180ee17ef2404e249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:58 UTC) from {username='slave'} at 127.0.0.1:34548
I20260619 16:53:58.242976 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 32706
I20260619 16:53:58.253893 669 tablet_replica.cc:330] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: stopping tablet replica
I20260619 16:53:58.254472 669 raft_consensus.cc:2229] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:58.255605 670 tablet_replica.cc:330] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3: stopping tablet replica
I20260619 16:53:58.256119 670 raft_consensus.cc:2229] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:53:58.256982 670 raft_consensus.cc:2258] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:58.257557 669 raft_consensus.cc:2258] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:58.259346 668 tablet_replica.cc:330] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: stopping tablet replica
I20260619 16:53:58.259735 668 raft_consensus.cc:2229] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:58.260262 668 raft_consensus.cc:2258] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:58.261315 669 ts_tablet_manager.cc:1905] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:58.262266 668 ts_tablet_manager.cc:1905] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:58.265239 670 ts_tablet_manager.cc:1905] T 90f0a856bac0479180ee17ef2404e249 P 80b04f35fe0c47318f84bf429e6aa0a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:58.268692 669 ts_tablet_manager.cc:1918] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:58.268930 669 log.cc:1196] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/90f0a856bac0479180ee17ef2404e249
I20260619 16:53:58.269639 669 ts_tablet_manager.cc:1939] T 90f0a856bac0479180ee17ef2404e249 P e7b49dfdde704845b3614c4c341b05e2: Deleting consensus metadata
I20260619 16:53:58.269833 668 ts_tablet_manager.cc:1918] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:58.269920 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.244808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.14.153.1:41111)
0619 16:53:58.245061 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:58.245071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:58.258582 (+ 13511us) server_negotiation.cc:407] Connection header received
0619 16:53:58.258813 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:58.258820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:58.258938 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:58.259072 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:58.259786 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.260843 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.262782 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.263416 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.263513 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:58.268014 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:58.268026 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:58.268047 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:58.268918 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:58.268978 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:58.269013 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:58.269092 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:58.269381 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 16:53:58.269516 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:53:58.270002 668 log.cc:1196] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/90f0a856bac0479180ee17ef2404e249
I20260619 16:53:58.270576 668 ts_tablet_manager.cc:1939] T 90f0a856bac0479180ee17ef2404e249 P 360bfadc4b304c71901dc439fa400b34: Deleting consensus metadata
I20260619 16:53:58.271317 360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:58.272099 32633 catalog_manager.cc:4841] TS e7b49dfdde704845b3614c4c341b05e2 (127.14.153.3:42311): tablet 90f0a856bac0479180ee17ef2404e249 (table test-table [id=6b244afe08f14557bf62c64b0f6141f4]) successfully deleted
I20260619 16:53:58.272852 32631 catalog_manager.cc:4841] TS 360bfadc4b304c71901dc439fa400b34 (127.14.153.2:39349): tablet 90f0a856bac0479180ee17ef2404e249 (table test-table [id=6b244afe08f14557bf62c64b0f6141f4]) successfully deleted
I20260619 16:53:58.379436 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 32706
I20260619 16:53:58.398828 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 369
W20260619 16:53:58.398940 32631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.153.1:41111 (error 108)
I20260619 16:53:58.400962 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.400217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41111 (local address 127.0.0.1:34576)
0619 16:53:58.400685 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:58.400804 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":250,"threads_started":1}
W20260619 16:53:58.401239 32631 catalog_manager.cc:4584] TS 80b04f35fe0c47318f84bf429e6aa0a3 (127.14.153.1:41111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90f0a856bac0479180ee17ef2404e249: Network error: Client connection negotiation failed: client connection to 127.14.153.1:41111: connect: Connection refused (error 111)
I20260619 16:53:58.406198 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.399779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.14.153.2:39349)
0619 16:53:58.399990 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:58.399998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:58.400103 (+ 105us) server_negotiation.cc:407] Connection header received
0619 16:53:58.400292 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:58.400296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:58.400375 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:58.400505 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:58.400948 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.401803 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.403168 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.403586 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.403718 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:58.404844 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:58.404853 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:58.404868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:58.405448 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:58.405493 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:58.405511 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:58.405643 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:58.405754 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 16:53:58.405859 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:53:58.406809 491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:58.454939 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.454328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41111 (local address 127.0.0.1:34586)
0619 16:53:58.454598 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:58.454795 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260619 16:53:58.456049 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.455687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41111 (local address 127.0.0.1:34592)
0619 16:53:58.455861 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:58.455924 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:53:58.496709 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 369
I20260619 16:53:58.513839 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 499
I20260619 16:53:58.516649 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.516230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41111 (local address 127.0.0.1:34596)
0619 16:53:58.516408 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:58.516487 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 16:53:58.517745 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.517353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41111 (local address 127.0.0.1:34612)
0619 16:53:58.517496 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:58.517583 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 16:53:58.521076 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.514589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.14.153.3:42311)
0619 16:53:58.514801 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:58.514805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:58.514825 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:53:58.514971 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:58.514975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:58.515111 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:58.515197 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:58.515588 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.516621 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.518320 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:58.518718 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:58.518813 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:58.519780 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:58.519788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:58.519803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:58.520361 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:58.520406 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:58.520424 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:58.520509 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:58.520663 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 16:53:58.520746 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 16:53:58.521733 621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:58.598767 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.598256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41111 (local address 127.0.0.1:34616)
0619 16:53:58.598475 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:58.598594 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 16:53:58.600039 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:58.599469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:41111 (local address 127.0.0.1:34628)
0619 16:53:58.599780 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:58.599871 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260619 16:53:58.611832 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 499
I20260619 16:53:58.628435 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 32616
I20260619 16:53:58.629588 32677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:53:58.740185 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 32616
2026-06-19T16:53:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2926 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:58 dist-test-slave-qb36 krb5kdc[680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:58 dist-test-slave-qb36 krb5kdc[680](info): set up 2 sockets
Jun 19 16:53:58 dist-test-slave-qb36 krb5kdc[680](info): commencing operation
krb5kdc: starting...
W20260619 16:53:59.847147 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:00 dist-test-slave-qb36 krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:00.098403 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:45843
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:00.429348 693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:00.429674 693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:00.429864 693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:00.437390 693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:00.437515 693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:00.437618 693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:00.437731 693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:00.437795 693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:00.448160 693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:00.449555 693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:00.450908 693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:00.462953 699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:00.463073 698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:00.463630 701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:00.463737 693 server_base.cc:1034] running on GCE node
I20260619 16:54:00.464537 693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:00.465884 693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:00.467124 693 hybrid_clock.cc:648] HybridClock initialized: now 1781888040467072 us; error 61 us; skew 500 ppm
I20260619 16:54:00.467487 693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:00 dist-test-slave-qb36 krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:00.470625 693 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:00.472114 693 webserver.cc:460] Webserver started at http://127.14.153.62:36393/ using document root <none> and password file <none>
I20260619 16:54:00.472818 693 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:00.472970 693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:00.473282 693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:00.476461 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "b80101140a614994909ab9b662d55c3b"
format_stamp: "Formatted at 2026-06-19 16:54:00 on dist-test-slave-qb36"
server_key: "e5c5ee9c4253a771c61d64f15c4e4473"
server_key_iv: "edab97bb23fa660b3aa49b736c449a38"
server_key_version: "encryptionkey@0"
I20260619 16:54:00.477210 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "b80101140a614994909ab9b662d55c3b"
format_stamp: "Formatted at 2026-06-19 16:54:00 on dist-test-slave-qb36"
server_key: "e5c5ee9c4253a771c61d64f15c4e4473"
server_key_iv: "edab97bb23fa660b3aa49b736c449a38"
server_key_version: "encryptionkey@0"
I20260619 16:54:00.482657 693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:54:00.486424 708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:00.487929 693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:00.488122 693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "b80101140a614994909ab9b662d55c3b"
format_stamp: "Formatted at 2026-06-19 16:54:00 on dist-test-slave-qb36"
server_key: "e5c5ee9c4253a771c61d64f15c4e4473"
server_key_iv: "edab97bb23fa660b3aa49b736c449a38"
server_key_version: "encryptionkey@0"
I20260619 16:54:00.488343 693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:00.513269 693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:00.517952 693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:00.518263 693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:00.537262 693 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:45843
I20260619 16:54:00.537282 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:45843 every 8 connection(s)
I20260619 16:54:00.538667 693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:00.542451 760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:00.543377 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 693
I20260619 16:54:00.543550 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:00.544160 14948 external_mini_cluster.cc:1428] Setting key cfefc4b668798d5bec374edb76646e59
I20260619 16:54:00.557255 760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b: Bootstrap starting.
Jun 19 16:54:00 dist-test-slave-qb36 krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:00.561082 760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:00.562211 760 log.cc:824] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:00.565557 760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b: No bootstrap required, opened a new log
I20260619 16:54:00.569420 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:00.546157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.14.153.62:45843)
0619 16:54:00.547068 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:00.547086 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:00.547139 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:54:00.548811 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:00.548838 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:00.549271 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:00.549637 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:00.551546 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:00.552884 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:00.555158 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:00.555867 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:00.556008 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:00.560850 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:00.560859 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:00.560875 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:00.560917 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:00.564666 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:00.565431 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:00.565440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:00.565449 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:00.565697 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:00.566071 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:00.566077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:00.566083 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:00.566791 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:00.567137 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:00.567493 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 16:54:00.567980 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":388,"threads_started":1}
I20260619 16:54:00.575163 760 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80101140a614994909ab9b662d55c3b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45843 } }
I20260619 16:54:00.575555 760 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:00.575637 760 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80101140a614994909ab9b662d55c3b, State: Initialized, Role: FOLLOWER
I20260619 16:54:00.576143 760 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80101140a614994909ab9b662d55c3b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45843 } }
I20260619 16:54:00.576361 760 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:00.576460 760 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:00.576592 760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:00.579099 760 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80101140a614994909ab9b662d55c3b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45843 } }
I20260619 16:54:00.579623 760 leader_election.cc:302] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80101140a614994909ab9b662d55c3b; no voters:
I20260619 16:54:00.580207 760 leader_election.cc:288] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:00.580364 765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:00.581457 765 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [term 1 LEADER]: Becoming Leader. State: Replica: b80101140a614994909ab9b662d55c3b, State: Running, Role: LEADER
I20260619 16:54:00.582113 765 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80101140a614994909ab9b662d55c3b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45843 } }
I20260619 16:54:00.582571 760 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:00.585304 766 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b80101140a614994909ab9b662d55c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80101140a614994909ab9b662d55c3b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45843 } } }
I20260619 16:54:00.585330 767 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80101140a614994909ab9b662d55c3b. Latest consensus state: current_term: 1 leader_uuid: "b80101140a614994909ab9b662d55c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80101140a614994909ab9b662d55c3b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 45843 } } }
I20260619 16:54:00.585955 767 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:00.585960 766 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:00.597678 770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:00.602088 770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:00.610350 770 catalog_manager.cc:1347] Generated new cluster ID: 98eed038a275462685e5b1d6fdc80d66
I20260619 16:54:00.610481 770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:00.629446 770 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:00.631392 770 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:00.645033 770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b: Generated new TSK 0
I20260619 16:54:00.646190 770 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:54:00.703926 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45843
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:01.039525 786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:01.039819 786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:01.039995 786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:01.047402 786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:01.047551 786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:01.047726 786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:54:01.057499 786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:01.058816 786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:01.060158 786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:01.073138 791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:01.073035 792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:01.073057 794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:01.073951 786 server_base.cc:1034] running on GCE node
I20260619 16:54:01.074645 786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:01.075673 786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:01.076923 786 hybrid_clock.cc:648] HybridClock initialized: now 1781888041076874 us; error 46 us; skew 500 ppm
I20260619 16:54:01.077275 786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:01 dist-test-slave-qb36 krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:01.080643 786 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:54:01.082234 786 webserver.cc:460] Webserver started at http://127.14.153.1:33693/ using document root <none> and password file <none>
I20260619 16:54:01.083134 786 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:01.083273 786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:01.083601 786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:01.086640 786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "c1c96c523f3f44f1a19c3d8685431da7"
format_stamp: "Formatted at 2026-06-19 16:54:01 on dist-test-slave-qb36"
server_key: "4c5bb5c047fa0e6157aed00b076767a7"
server_key_iv: "44fe6191a39f9a269d70f712cf97d3b5"
server_key_version: "encryptionkey@0"
I20260619 16:54:01.087438 786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "c1c96c523f3f44f1a19c3d8685431da7"
format_stamp: "Formatted at 2026-06-19 16:54:01 on dist-test-slave-qb36"
server_key: "4c5bb5c047fa0e6157aed00b076767a7"
server_key_iv: "44fe6191a39f9a269d70f712cf97d3b5"
server_key_version: "encryptionkey@0"
I20260619 16:54:01.092872 786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:54:01.096779 801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:01.098161 786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:54:01.098340 786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "c1c96c523f3f44f1a19c3d8685431da7"
format_stamp: "Formatted at 2026-06-19 16:54:01 on dist-test-slave-qb36"
server_key: "4c5bb5c047fa0e6157aed00b076767a7"
server_key_iv: "44fe6191a39f9a269d70f712cf97d3b5"
server_key_version: "encryptionkey@0"
I20260619 16:54:01.098532 786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:01.115326 786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:01.119510 786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:01.119812 786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:01.120784 786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:01.122134 786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:01.122210 786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:01.122356 786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:01.122418 786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:01.156304 786 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:37753
I20260619 16:54:01.156353 913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:37753 every 8 connection(s)
I20260619 16:54:01.157650 786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:54:01.167938 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 786
I20260619 16:54:01.168164 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:54:01.168603 14948 external_mini_cluster.cc:1428] Setting key 66719fea6dd0244b7d84fa212d4d4d8d
Jun 19 16:54:01 dist-test-slave-qb36 krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:01.177270 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:01.160191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:45353 (local address 127.14.153.62:45843)
0619 16:54:01.160635 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:01.160640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:01.161494 (+ 854us) server_negotiation.cc:407] Connection header received
0619 16:54:01.163402 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:01.163408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:01.163496 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:01.163610 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:01.165068 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:01.165794 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:01.167394 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:01.167813 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:01.167951 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:01.171462 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:01.171467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:01.171470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:01.171503 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:01.174078 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:01.174859 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:01.174863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:01.174865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:01.174987 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:01.175447 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:01.175453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:01.175455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:01.175751 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:01.175912 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:01.176661 (+ 749us) server_negotiation.cc:299] Negotiation successful
0619 16:54:01.176912 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260619 16:54:01.178308 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:01.160640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45843 (local address 127.14.153.1:45353)
0619 16:54:01.161340 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:01.161383 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:01.163134 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:01.163846 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:01.163854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:01.164357 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:01.164902 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:01.164918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:01.165966 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:01.165970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:01.167245 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:01.167256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:01.168041 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:01.168049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:01.168342 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:01.169058 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:01.169080 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:01.171251 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:01.174250 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:01.174260 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:01.174279 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:01.174724 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:01.175111 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:01.175116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:01.175119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:01.175328 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:01.175883 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:01.175891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:01.176271 (+ 380us) client_negotiation.cc:769] Sending connection context
0619 16:54:01.176539 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 16:54:01.177122 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":274,"threads_started":1}
I20260619 16:54:01.179997 914 heartbeater.cc:347] Connected to a master server at 127.14.153.62:45843
I20260619 16:54:01.180306 914 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:01.181075 914 heartbeater.cc:511] Master 127.14.153.62:45843 requested a full tablet report, sending...
I20260619 16:54:01.183539 725 ts_manager.cc:194] Registered new tserver with Master: c1c96c523f3f44f1a19c3d8685431da7 (127.14.153.1:37753)
I20260619 16:54:01.185813 725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:45353
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:54:01.226233 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45843
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:01.565891 920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:01.566184 920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:01.566278 920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:01.573567 920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:01.573685 920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:01.573801 920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:54:01.583781 920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:01.585106 920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:01.586360 920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:01.601137 925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:01.601269 926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:01.601243 928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:01.601872 920 server_base.cc:1034] running on GCE node
I20260619 16:54:01.602527 920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:01.603708 920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:01.604934 920 hybrid_clock.cc:648] HybridClock initialized: now 1781888041604853 us; error 61 us; skew 500 ppm
I20260619 16:54:01.605312 920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:01 dist-test-slave-qb36 krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:01.608696 920 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:54:01.610365 920 webserver.cc:460] Webserver started at http://127.14.153.2:37835/ using document root <none> and password file <none>
I20260619 16:54:01.611158 920 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:01.611302 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:01.611644 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:01.614720 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "270db30b498a43e7ad04033cd6bcba2f"
format_stamp: "Formatted at 2026-06-19 16:54:01 on dist-test-slave-qb36"
server_key: "583313d8096bde0fbf5c59f9cfe85f0c"
server_key_iv: "27dc15c9b1922f3537bab9e27b0d4a44"
server_key_version: "encryptionkey@0"
I20260619 16:54:01.615595 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "270db30b498a43e7ad04033cd6bcba2f"
format_stamp: "Formatted at 2026-06-19 16:54:01 on dist-test-slave-qb36"
server_key: "583313d8096bde0fbf5c59f9cfe85f0c"
server_key_iv: "27dc15c9b1922f3537bab9e27b0d4a44"
server_key_version: "encryptionkey@0"
I20260619 16:54:01.620929 920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:54:01.624312 935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:01.625658 920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:01.625828 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "270db30b498a43e7ad04033cd6bcba2f"
format_stamp: "Formatted at 2026-06-19 16:54:01 on dist-test-slave-qb36"
server_key: "583313d8096bde0fbf5c59f9cfe85f0c"
server_key_iv: "27dc15c9b1922f3537bab9e27b0d4a44"
server_key_version: "encryptionkey@0"
I20260619 16:54:01.626025 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:01.671626 920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:01.676337 920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:01.676671 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:01.677757 920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:01.679239 920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:01.679342 920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:01.679481 920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:01.679548 920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:01.712682 920 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:45847
I20260619 16:54:01.712716 1047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:45847 every 8 connection(s)
I20260619 16:54:01.714046 920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:54:01.721966 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 920
I20260619 16:54:01.722141 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:54:01.722568 14948 external_mini_cluster.cc:1428] Setting key 721939f22341f425957673d3e5c27526
Jun 19 16:54:01 dist-test-slave-qb36 krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:01.734772 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:01.716747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:58851 (local address 127.14.153.62:45843)
0619 16:54:01.717218 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:01.717225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:01.718178 (+ 953us) server_negotiation.cc:407] Connection header received
0619 16:54:01.720047 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:01.720052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:01.720122 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:01.720220 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:01.721711 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:01.722520 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:01.724521 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:01.724953 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:01.725092 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:01.728557 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:01.728562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:01.728566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:01.728599 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:01.731356 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:01.732227 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:01.732237 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:01.732240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:01.732371 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:01.732920 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:01.732924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:01.732927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:01.733218 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:01.733340 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:01.734272 (+ 932us) server_negotiation.cc:299] Negotiation successful
0619 16:54:01.734489 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
I20260619 16:54:01.736330 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:01.717303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45843 (local address 127.14.153.2:58851)
0619 16:54:01.718001 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:01.718052 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:01.719794 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:01.720417 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:01.720426 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:01.720927 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:01.721512 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:01.721530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:01.722703 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:01.722709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:01.724365 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:01.724381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:01.725185 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:01.725192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:01.725424 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:01.726096 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:01.726118 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:01.728322 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:01.731498 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:01.731570 (+ 72us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:01.731586 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:01.732075 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:01.732489 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:01.732497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:01.732503 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:01.732771 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:01.733350 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:01.733361 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:01.733837 (+ 476us) client_negotiation.cc:769] Sending connection context
0619 16:54:01.734147 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 16:54:01.734745 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":259,"threads_started":1}
I20260619 16:54:01.737854 1048 heartbeater.cc:347] Connected to a master server at 127.14.153.62:45843
I20260619 16:54:01.738183 1048 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:01.739099 1048 heartbeater.cc:511] Master 127.14.153.62:45843 requested a full tablet report, sending...
I20260619 16:54:01.740721 725 ts_manager.cc:194] Registered new tserver with Master: 270db30b498a43e7ad04033cd6bcba2f (127.14.153.2:45847)
I20260619 16:54:01.741559 725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:58851
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:54:01.778497 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45843
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:02.111759 1054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:02.112134 1054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:02.112304 1054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:02.119972 1054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:02.120139 1054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:02.120321 1054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:54:02.130836 1054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:02.132225 1054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:02.133558 1054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:02.147300 1059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:02.147269 1062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:02.147379 1060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:02.148005 1054 server_base.cc:1034] running on GCE node
I20260619 16:54:02.148633 1054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:02.149644 1054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:02.150875 1054 hybrid_clock.cc:648] HybridClock initialized: now 1781888042150804 us; error 64 us; skew 500 ppm
I20260619 16:54:02.151261 1054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:02 dist-test-slave-qb36 krb5kdc[680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:02.154541 1054 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:54:02.156257 1054 webserver.cc:460] Webserver started at http://127.14.153.3:46067/ using document root <none> and password file <none>
I20260619 16:54:02.156975 1054 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:02.157102 1054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:02.157434 1054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:02.160398 1054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "67e41d10542f414d9c1a7b2afb523ea9"
format_stamp: "Formatted at 2026-06-19 16:54:02 on dist-test-slave-qb36"
server_key: "0f11a08e7325f1457f36ce65ff06a23b"
server_key_iv: "ecafe6c1c7b0fea6b830b5f1267c25a3"
server_key_version: "encryptionkey@0"
I20260619 16:54:02.161161 1054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "67e41d10542f414d9c1a7b2afb523ea9"
format_stamp: "Formatted at 2026-06-19 16:54:02 on dist-test-slave-qb36"
server_key: "0f11a08e7325f1457f36ce65ff06a23b"
server_key_iv: "ecafe6c1c7b0fea6b830b5f1267c25a3"
server_key_version: "encryptionkey@0"
I20260619 16:54:02.166518 1054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:54:02.169917 1069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:02.171332 1054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:02.171494 1054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "67e41d10542f414d9c1a7b2afb523ea9"
format_stamp: "Formatted at 2026-06-19 16:54:02 on dist-test-slave-qb36"
server_key: "0f11a08e7325f1457f36ce65ff06a23b"
server_key_iv: "ecafe6c1c7b0fea6b830b5f1267c25a3"
server_key_version: "encryptionkey@0"
I20260619 16:54:02.171693 1054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:02.188992 914 heartbeater.cc:503] Master 127.14.153.62:45843 was elected leader, sending a full tablet report...
I20260619 16:54:02.193694 1054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:02.198201 1054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:02.198490 1054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:02.199564 1054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:02.200866 1054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:02.200963 1054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:02.201104 1054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:02.201180 1054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:02.234644 1054 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:38921
I20260619 16:54:02.234742 1181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:38921 every 8 connection(s)
I20260619 16:54:02.236069 1054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:54:02.237922 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 1054
I20260619 16:54:02.238101 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:54:02.238554 14948 external_mini_cluster.cc:1428] Setting key 253b8aa4590fdb6f551ce44fd52c8811
Jun 19 16:54:02 dist-test-slave-qb36 krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:02.256462 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.238906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:33853 (local address 127.14.153.62:45843)
0619 16:54:02.239429 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.239435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.240518 (+ 1083us) server_negotiation.cc:407] Connection header received
0619 16:54:02.242547 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.242551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.242624 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.242748 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:02.244270 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.245027 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.246548 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.247092 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.247197 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.250512 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:02.250517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:02.250520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:02.250553 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:02.253653 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.254449 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.254453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.254455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.254579 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.254925 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.254927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.254929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.255248 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:02.255366 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.256019 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.256186 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":251,"threads_started":1}
I20260619 16:54:02.257395 1185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.239543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:45843 (local address 127.14.153.3:33853)
0619 16:54:02.240348 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:02.240387 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:02.242304 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:02.242925 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:02.242932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:02.243497 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:02.244121 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.244140 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.245185 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.245190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.246415 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.246427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.247312 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.247317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.247529 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.248119 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:02.248147 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:02.250317 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:02.253805 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:02.253817 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:02.253834 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:02.254350 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:02.254668 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:02.254671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:02.254673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:02.254840 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:02.255351 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:02.255357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:02.255713 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 16:54:02.255944 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 16:54:02.256320 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":285,"threads_started":1}
I20260619 16:54:02.258616 1182 heartbeater.cc:347] Connected to a master server at 127.14.153.62:45843
I20260619 16:54:02.258880 1182 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:02.259582 1182 heartbeater.cc:511] Master 127.14.153.62:45843 requested a full tablet report, sending...
I20260619 16:54:02.260912 725 ts_manager.cc:194] Registered new tserver with Master: 67e41d10542f414d9c1a7b2afb523ea9 (127.14.153.3:38921)
I20260619 16:54:02.261698 725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:33853
I20260619 16:54:02.265935 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:54:02.283293 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.270098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.14.153.62:45843)
0619 16:54:02.270323 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.270331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.270565 (+ 234us) server_negotiation.cc:407] Connection header received
0619 16:54:02.270709 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.270713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.270771 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.270850 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:02.272337 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.273026 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.274301 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.274725 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.274861 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.277655 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:02.277660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:02.277663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:02.277692 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:02.280318 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.281196 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.281200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.281202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.281319 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.281722 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.281725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.281726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.282080 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:02.282298 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.282713 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.282878 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:54:02.287921 725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:54:02.295203 725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:54:02.321187 1195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.309372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:38921 (local address 127.0.0.1:35920)
0619 16:54:02.312268 (+ 2896us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:02.312289 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:02.312413 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:02.313653 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:02.313657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:02.313683 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:02.314524 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.314535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.316247 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.316252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.318382 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.318395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.320035 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.320041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.320254 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.320312 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 16:54:02.320448 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 16:54:02.320663 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2761,"mutex_wait_us":155,"thread_start_us":1347,"threads_started":1}
I20260619 16:54:02.322006 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.308559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:45847 (local address 127.0.0.1:42314)
0619 16:54:02.310195 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:02.310241 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:02.310407 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:02.313402 (+ 2995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:02.313410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:02.313442 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:02.314522 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.314535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.316446 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.316455 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.319073 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.319088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.321148 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.321157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.321327 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.321358 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:54:02.321451 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:54:02.321590 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":130,"thread_start_us":3416,"threads_started":1}
I20260619 16:54:02.322088 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.310039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.14.153.3:38921)
0619 16:54:02.313035 (+ 2996us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.313042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.313070 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:54:02.313146 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.313151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.313372 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.313581 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:02.314772 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.316084 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.318549 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.319831 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.321259 (+ 1428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.321482 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.321552 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.321746 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2796,"thread_start_us":1394,"threads_started":1}
I20260619 16:54:02.322237 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.309903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.14.153.1:37753)
0619 16:54:02.312695 (+ 2792us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.312705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.312734 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:54:02.312841 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.312849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.313129 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.313352 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:02.314328 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.315886 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.319271 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.320410 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.320534 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.320773 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.321708 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.321900 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2572,"thread_start_us":2429,"threads_started":1}
I20260619 16:54:02.322582 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.309262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:37753 (local address 127.0.0.1:54924)
0619 16:54:02.311075 (+ 1813us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:02.311098 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:02.311224 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:02.313376 (+ 2152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:02.313384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:02.313418 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:02.314144 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.314157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.316061 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.316067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.319078 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.319088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.320922 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.320928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.321162 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.321203 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:54:02.321295 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:54:02.321433 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1620,"mutex_wait_us":92,"thread_start_us":831,"threads_started":1}
I20260619 16:54:02.323658 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.308868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.14.153.2:45847)
0619 16:54:02.312882 (+ 4014us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.312891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.312913 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:54:02.312993 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.312999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.313201 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.313518 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:02.314735 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.316279 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.319232 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.320637 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.320767 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.320975 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.323252 (+ 2277us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.323411 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3864,"thread_start_us":3770,"threads_started":1}
I20260619 16:54:02.326335 1117 tablet_service.cc:1467] Processing CreateTablet for tablet ad380f7e7f79432499108751736818ba (DEFAULT_TABLE table=test-table [id=14d888e8f394492a85951ea5122e8e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:02.326336 849 tablet_service.cc:1467] Processing CreateTablet for tablet ad380f7e7f79432499108751736818ba (DEFAULT_TABLE table=test-table [id=14d888e8f394492a85951ea5122e8e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:02.327163 983 tablet_service.cc:1467] Processing CreateTablet for tablet ad380f7e7f79432499108751736818ba (DEFAULT_TABLE table=test-table [id=14d888e8f394492a85951ea5122e8e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:02.327762 1117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad380f7e7f79432499108751736818ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:02.327834 849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad380f7e7f79432499108751736818ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:02.328433 983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad380f7e7f79432499108751736818ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:02.341728 1201 tablet_bootstrap.cc:492] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: Bootstrap starting.
I20260619 16:54:02.343961 1202 tablet_bootstrap.cc:492] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: Bootstrap starting.
I20260619 16:54:02.346163 1201 tablet_bootstrap.cc:654] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:02.347620 1201 log.cc:824] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:02.347916 1202 tablet_bootstrap.cc:654] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:02.349295 1202 log.cc:824] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:02.351876 1203 tablet_bootstrap.cc:492] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: Bootstrap starting.
I20260619 16:54:02.352101 1201 tablet_bootstrap.cc:492] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: No bootstrap required, opened a new log
I20260619 16:54:02.353078 1201 ts_tablet_manager.cc:1397] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260619 16:54:02.355371 1203 tablet_bootstrap.cc:654] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:02.356707 1203 log.cc:824] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:02.358088 1202 tablet_bootstrap.cc:492] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: No bootstrap required, opened a new log
I20260619 16:54:02.358404 1202 ts_tablet_manager.cc:1397] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260619 16:54:02.359463 1203 tablet_bootstrap.cc:492] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: No bootstrap required, opened a new log
I20260619 16:54:02.359725 1203 ts_tablet_manager.cc:1397] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 16:54:02.364406 1201 raft_consensus.cc:348] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } }
I20260619 16:54:02.364883 1201 raft_consensus.cc:374] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:02.365039 1201 raft_consensus.cc:729] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e41d10542f414d9c1a7b2afb523ea9, State: Initialized, Role: FOLLOWER
I20260619 16:54:02.365712 1201 consensus_queue.cc:260] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } }
I20260619 16:54:02.366853 1182 heartbeater.cc:503] Master 127.14.153.62:45843 was elected leader, sending a full tablet report...
I20260619 16:54:02.367661 1201 ts_tablet_manager.cc:1428] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260619 16:54:02.369398 1202 raft_consensus.cc:348] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } }
I20260619 16:54:02.369972 1202 raft_consensus.cc:374] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:02.370121 1202 raft_consensus.cc:729] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c96c523f3f44f1a19c3d8685431da7, State: Initialized, Role: FOLLOWER
I20260619 16:54:02.370340 1203 raft_consensus.cc:348] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } }
I20260619 16:54:02.370774 1203 raft_consensus.cc:374] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:02.370879 1203 raft_consensus.cc:729] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270db30b498a43e7ad04033cd6bcba2f, State: Initialized, Role: FOLLOWER
I20260619 16:54:02.370880 1202 consensus_queue.cc:260] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } }
I20260619 16:54:02.371628 1203 consensus_queue.cc:260] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } }
I20260619 16:54:02.373011 1202 ts_tablet_manager.cc:1428] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 16:54:02.374707 1048 heartbeater.cc:503] Master 127.14.153.62:45843 was elected leader, sending a full tablet report...
I20260619 16:54:02.375442 1203 ts_tablet_manager.cc:1428] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260619 16:54:02.410146 915 tablet.cc:2369] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:54:02.437542 1207 raft_consensus.cc:482] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:54:02.437820 1207 raft_consensus.cc:504] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } }
I20260619 16:54:02.439589 1207 leader_election.cc:288] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 270db30b498a43e7ad04033cd6bcba2f (127.14.153.2:45847), c1c96c523f3f44f1a19c3d8685431da7 (127.14.153.1:37753)
I20260619 16:54:02.446141 1185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.439914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:45847 (local address 127.14.153.3:33671)
0619 16:54:02.440311 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:02.440341 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:02.440502 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:02.440944 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:02.440948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:02.440993 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:02.441176 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.441189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.442383 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.442389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.443928 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.443941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.445327 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.445334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.445536 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.445574 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:54:02.445690 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 16:54:02.445838 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260619 16:54:02.446137 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.440136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:33671 (local address 127.14.153.2:45847)
0619 16:54:02.440349 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.440356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.440503 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:54:02.440669 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.440675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.440758 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.440874 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:02.441313 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.442241 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.444090 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.445170 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.445287 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.445343 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.445708 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.445810 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:54:02.446818 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.439986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:55139 (local address 127.14.153.1:37753)
0619 16:54:02.440196 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.440204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.440876 (+ 672us) server_negotiation.cc:407] Connection header received
0619 16:54:02.441057 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.441063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.441144 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.441265 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:02.441887 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.442757 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.444783 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.445787 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.445934 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.446008 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.446388 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.446568 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:54:02.447450 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.439954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:37753 (local address 127.14.153.3:55139)
0619 16:54:02.440766 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:02.440784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:02.440909 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:02.441312 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:02.441316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:02.441340 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:02.441731 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.441743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.442936 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.442943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.444572 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:02.444586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.445988 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.445994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:02.446184 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.446220 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:54:02.447016 (+ 796us) client_negotiation.cc:240] Negotiation successful
0619 16:54:02.447215 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":411,"threads_started":1}
I20260619 16:54:02.447454 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad380f7e7f79432499108751736818ba" candidate_uuid: "67e41d10542f414d9c1a7b2afb523ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270db30b498a43e7ad04033cd6bcba2f" is_pre_election: true
I20260619 16:54:02.448042 1003 raft_consensus.cc:2454] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e41d10542f414d9c1a7b2afb523ea9 in term 0.
I20260619 16:54:02.448566 869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad380f7e7f79432499108751736818ba" candidate_uuid: "67e41d10542f414d9c1a7b2afb523ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1c96c523f3f44f1a19c3d8685431da7" is_pre_election: true
I20260619 16:54:02.448804 1071 leader_election.cc:302] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 270db30b498a43e7ad04033cd6bcba2f, 67e41d10542f414d9c1a7b2afb523ea9; no voters:
I20260619 16:54:02.449144 869 raft_consensus.cc:2454] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e41d10542f414d9c1a7b2afb523ea9 in term 0.
I20260619 16:54:02.449234 1207 raft_consensus.cc:2781] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:54:02.449384 1207 raft_consensus.cc:482] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:54:02.449488 1207 raft_consensus.cc:3037] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:02.451853 1207 raft_consensus.cc:504] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } }
I20260619 16:54:02.452400 1207 leader_election.cc:288] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 270db30b498a43e7ad04033cd6bcba2f (127.14.153.2:45847), c1c96c523f3f44f1a19c3d8685431da7 (127.14.153.1:37753)
I20260619 16:54:02.452908 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad380f7e7f79432499108751736818ba" candidate_uuid: "67e41d10542f414d9c1a7b2afb523ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270db30b498a43e7ad04033cd6bcba2f"
I20260619 16:54:02.452908 869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad380f7e7f79432499108751736818ba" candidate_uuid: "67e41d10542f414d9c1a7b2afb523ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1c96c523f3f44f1a19c3d8685431da7"
I20260619 16:54:02.453121 1003 raft_consensus.cc:3037] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:02.453143 869 raft_consensus.cc:3037] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:02.456439 869 raft_consensus.cc:2454] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e41d10542f414d9c1a7b2afb523ea9 in term 1.
I20260619 16:54:02.456441 1003 raft_consensus.cc:2454] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e41d10542f414d9c1a7b2afb523ea9 in term 1.
I20260619 16:54:02.456912 1071 leader_election.cc:302] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 270db30b498a43e7ad04033cd6bcba2f, 67e41d10542f414d9c1a7b2afb523ea9, c1c96c523f3f44f1a19c3d8685431da7; no voters:
I20260619 16:54:02.457254 1207 raft_consensus.cc:2781] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:02.457693 1207 raft_consensus.cc:686] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 1 LEADER]: Becoming Leader. State: Replica: 67e41d10542f414d9c1a7b2afb523ea9, State: Running, Role: LEADER
I20260619 16:54:02.458168 1207 consensus_queue.cc:237] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } }
I20260619 16:54:02.462694 724 catalog_manager.cc:5495] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67e41d10542f414d9c1a7b2afb523ea9 (127.14.153.3). New cstate: current_term: 1 leader_uuid: "67e41d10542f414d9c1a7b2afb523ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e41d10542f414d9c1a7b2afb523ea9" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 38921 } health_report { overall_health: HEALTHY } } }
W20260619 16:54:02.466140 1049 tablet.cc:2369] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:54:02.487749 1183 tablet.cc:2369] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:54:02.515317 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.509812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.14.153.3:38921)
0619 16:54:02.510010 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.510015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.510161 (+ 146us) server_negotiation.cc:407] Connection header received
0619 16:54:02.510314 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.510317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.510392 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.510526 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:02.510925 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.511673 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.513147 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.513555 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.513659 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.514214 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:02.514712 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:02.514842 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.514995 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.515092 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:54:02.523797 869 raft_consensus.cc:1261] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [term 1 FOLLOWER]: Refusing update from remote peer 67e41d10542f414d9c1a7b2afb523ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:02.523809 1003 raft_consensus.cc:1261] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [term 1 FOLLOWER]: Refusing update from remote peer 67e41d10542f414d9c1a7b2afb523ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:02.524807 1207 consensus_queue.cc:1035] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "270db30b498a43e7ad04033cd6bcba2f" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:02.525111 1211 consensus_queue.cc:1035] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1c96c523f3f44f1a19c3d8685431da7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:02.538663 1215 mvcc.cc:205] Tried to move back new op lower bound from 7298613422166310912 to 7298613421912465408. Current Snapshot: MvccSnapshot[applied={T|T < 7298613422166310912}]
I20260619 16:54:02.568717 724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:54:02.569026 724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:54:02.572753 724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b80101140a614994909ab9b662d55c3b: Sending DeleteTablet for 3 replicas of tablet ad380f7e7f79432499108751736818ba
I20260619 16:54:02.574054 849 tablet_service.cc:1514] Processing DeleteTablet for tablet ad380f7e7f79432499108751736818ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:02 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:54924
I20260619 16:54:02.574054 1117 tablet_service.cc:1514] Processing DeleteTablet for tablet ad380f7e7f79432499108751736818ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:02 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:35920
I20260619 16:54:02.574045 983 tablet_service.cc:1514] Processing DeleteTablet for tablet ad380f7e7f79432499108751736818ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:02 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:42314
I20260619 16:54:02.578554 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 786
I20260619 16:54:02.599953 1224 tablet_replica.cc:330] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: stopping tablet replica
I20260619 16:54:02.600642 1223 tablet_replica.cc:330] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: stopping tablet replica
I20260619 16:54:02.601025 1225 tablet_replica.cc:330] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: stopping tablet replica
I20260619 16:54:02.601471 1225 raft_consensus.cc:2229] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:54:02.601871 1223 raft_consensus.cc:2229] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:54:02.601476 1224 raft_consensus.cc:2229] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:54:02.602111 1225 raft_consensus.cc:2258] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:02.602914 1224 raft_consensus.cc:2258] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:02.603397 1223 raft_consensus.cc:2258] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:02.604933 1225 ts_tablet_manager.cc:1905] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:02.605484 1223 ts_tablet_manager.cc:1905] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 16:54:02 dist-test-slave-qb36 krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:54:02.612313 1225 ts_tablet_manager.cc:1918] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:02.612485 1223 ts_tablet_manager.cc:1918] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:02.612609 1225 log.cc:1196] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/ad380f7e7f79432499108751736818ba
I20260619 16:54:02.613289 1225 ts_tablet_manager.cc:1939] T ad380f7e7f79432499108751736818ba P 270db30b498a43e7ad04033cd6bcba2f: Deleting consensus metadata
I20260619 16:54:02.614847 1224 ts_tablet_manager.cc:1905] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:02.615625 710 catalog_manager.cc:4841] TS 270db30b498a43e7ad04033cd6bcba2f (127.14.153.2:45847): tablet ad380f7e7f79432499108751736818ba (table test-table [id=14d888e8f394492a85951ea5122e8e5a]) successfully deleted
I20260619 16:54:02.616356 1223 log.cc:1196] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/ad380f7e7f79432499108751736818ba
I20260619 16:54:02.617040 1223 ts_tablet_manager.cc:1939] T ad380f7e7f79432499108751736818ba P c1c96c523f3f44f1a19c3d8685431da7: Deleting consensus metadata
I20260619 16:54:02.619233 709 catalog_manager.cc:4841] TS c1c96c523f3f44f1a19c3d8685431da7 (127.14.153.1:37753): tablet ad380f7e7f79432499108751736818ba (table test-table [id=14d888e8f394492a85951ea5122e8e5a]) successfully deleted
I20260619 16:54:02.622008 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.580213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.14.153.1:37753)
0619 16:54:02.580460 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.580469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.590101 (+ 9632us) server_negotiation.cc:407] Connection header received
0619 16:54:02.590503 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.590514 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.590657 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.590766 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:02.593760 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.595653 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.601693 (+ 6040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.603172 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.603299 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.613610 (+ 10311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:02.613630 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:02.613651 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:02.613703 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:02.618228 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.619550 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.619562 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.619566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.619905 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.620404 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.620409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.620411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.620775 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:02.621087 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.621414 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.621638 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:54:02.622716 909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:02.623262 1224 ts_tablet_manager.cc:1918] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:02.623594 1224 log.cc:1196] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/ad380f7e7f79432499108751736818ba
I20260619 16:54:02.624430 1224 ts_tablet_manager.cc:1939] T ad380f7e7f79432499108751736818ba P 67e41d10542f414d9c1a7b2afb523ea9: Deleting consensus metadata
I20260619 16:54:02.626873 711 catalog_manager.cc:4841] TS 67e41d10542f414d9c1a7b2afb523ea9 (127.14.153.3:38921): tablet ad380f7e7f79432499108751736818ba (table test-table [id=14d888e8f394492a85951ea5122e8e5a]) successfully deleted
I20260619 16:54:02.714413 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 786
I20260619 16:54:02.732376 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 920
Jun 19 16:54:02 dist-test-slave-qb36 krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:54:02.750669 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.734410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.14.153.2:45847)
0619 16:54:02.734585 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.734592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.734919 (+ 327us) server_negotiation.cc:407] Connection header received
0619 16:54:02.735155 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.735161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.735244 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.735352 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:02.736679 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.737480 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.738864 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.739384 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.739514 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.743924 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:02.743939 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:02.743961 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:02.744012 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:02.747808 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.748555 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.748565 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.748569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.748723 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.749106 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.749111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.749115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.749589 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:02.749907 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.750067 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.750296 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:54:02.751669 1043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:02.842207 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 920
I20260619 16:54:02.860244 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 1054
Jun 19 16:54:02 dist-test-slave-qb36 krb5kdc[680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:54:02.879612 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:02.861245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.14.153.3:38921)
0619 16:54:02.861463 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:02.861474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:02.861498 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:54:02.861568 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:02.861573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:02.861664 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:02.861784 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:02.863183 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.864454 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.865832 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:02.866504 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:02.866638 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:02.871193 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:02.871213 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:02.871233 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:02.871278 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:02.875874 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.876989 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.876997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.876999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.877187 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:02.877694 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:02.877698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:02.877701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:02.878106 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:02.878357 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:02.878880 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 16:54:02.879156 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:54:02.880380 1177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:03.007525 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 1054
I20260619 16:54:03.030368 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 693
I20260619 16:54:03.031544 755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:03.142959 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 693
2026-06-19T16:54:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4403 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T16:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:03Z Disabled control of system clock
I20260619 16:54:03.220337 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:33679
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:33679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:54:03.557171 1234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:03.557476 1234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:03.557631 1234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:03.565073 1234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:03.565191 1234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:03.565291 1234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:03.565376 1234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:03.565459 1234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:03.575089 1234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:33679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:33679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:03.576499 1234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:03.578428 1234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:03.590400 1240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:03.590493 1242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:03.590468 1239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:03.591885 1234 server_base.cc:1034] running on GCE node
I20260619 16:54:03.592746 1234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:03.594298 1234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:03.595518 1234 hybrid_clock.cc:648] HybridClock initialized: now 1781888043595445 us; error 62 us; skew 500 ppm
I20260619 16:54:03.595911 1234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:54:03.598224 1234 webserver.cc:460] Webserver started at http://127.14.153.62:37661/ using document root <none> and password file <none>
I20260619 16:54:03.598928 1234 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:03.599077 1234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:03.599433 1234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:03.602317 1234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "d6e3aad600c24dda88fdbad582a57f86"
format_stamp: "Formatted at 2026-06-19 16:54:03 on dist-test-slave-qb36"
server_key: "a6590a7304f891e5bc8f443ff380d7a4"
server_key_iv: "0896c198c26011d69c058f26a4b2e171"
server_key_version: "encryptionkey@0"
I20260619 16:54:03.603281 1234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "d6e3aad600c24dda88fdbad582a57f86"
format_stamp: "Formatted at 2026-06-19 16:54:03 on dist-test-slave-qb36"
server_key: "a6590a7304f891e5bc8f443ff380d7a4"
server_key_iv: "0896c198c26011d69c058f26a4b2e171"
server_key_version: "encryptionkey@0"
I20260619 16:54:03.609362 1234 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:54:03.612865 1248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:03.614410 1234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:54:03.614571 1234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "d6e3aad600c24dda88fdbad582a57f86"
format_stamp: "Formatted at 2026-06-19 16:54:03 on dist-test-slave-qb36"
server_key: "a6590a7304f891e5bc8f443ff380d7a4"
server_key_iv: "0896c198c26011d69c058f26a4b2e171"
server_key_version: "encryptionkey@0"
I20260619 16:54:03.614774 1234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:03.640857 1234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:03.641698 1234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:03.641945 1234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:03.661285 1234 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:33679
I20260619 16:54:03.661309 1299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:33679 every 8 connection(s)
I20260619 16:54:03.662703 1234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:03.664758 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 1234
I20260619 16:54:03.664947 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:03.665396 14948 external_mini_cluster.cc:1428] Setting key 8c7320592ed2bbcf96a56e15d9aafd8e
I20260619 16:54:03.666965 1300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:03.679927 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:03.667407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.14.153.62:33679)
0619 16:54:03.668391 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:03.668402 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:03.668442 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:54:03.669978 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:03.669998 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:03.670478 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:03.670810 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:03.671438 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:03.672805 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:03.674789 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:03.675632 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:03.675794 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:03.677184 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:03.677201 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:03.677218 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:03.677863 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:03.677944 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:03.677970 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:03.678082 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:03.678401 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:54:03.678844 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":383,"threads_started":1}
I20260619 16:54:03.680922 1300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86: Bootstrap starting.
I20260619 16:54:03.684331 1300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:03.685387 1300 log.cc:824] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:03.688300 1300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86: No bootstrap required, opened a new log
I20260619 16:54:03.696339 1300 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e3aad600c24dda88fdbad582a57f86" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 33679 } }
I20260619 16:54:03.696774 1300 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:03.696888 1300 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e3aad600c24dda88fdbad582a57f86, State: Initialized, Role: FOLLOWER
I20260619 16:54:03.697520 1300 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e3aad600c24dda88fdbad582a57f86" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 33679 } }
I20260619 16:54:03.697773 1300 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:03.697912 1300 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:03.698087 1300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:03.700421 1300 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e3aad600c24dda88fdbad582a57f86" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 33679 } }
I20260619 16:54:03.700951 1300 leader_election.cc:302] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6e3aad600c24dda88fdbad582a57f86; no voters:
I20260619 16:54:03.701615 1300 leader_election.cc:288] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:03.701776 1305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:03.703087 1305 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [term 1 LEADER]: Becoming Leader. State: Replica: d6e3aad600c24dda88fdbad582a57f86, State: Running, Role: LEADER
I20260619 16:54:03.703619 1305 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e3aad600c24dda88fdbad582a57f86" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 33679 } }
I20260619 16:54:03.704042 1300 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:03.706316 1307 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6e3aad600c24dda88fdbad582a57f86. Latest consensus state: current_term: 1 leader_uuid: "d6e3aad600c24dda88fdbad582a57f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e3aad600c24dda88fdbad582a57f86" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 33679 } } }
I20260619 16:54:03.706753 1307 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:03.706719 1306 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6e3aad600c24dda88fdbad582a57f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e3aad600c24dda88fdbad582a57f86" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 33679 } } }
I20260619 16:54:03.707159 1306 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:03.708879 1314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:03.713585 1314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:03.723007 1314 catalog_manager.cc:1347] Generated new cluster ID: e34c5dd6f7164d85aad110819feb6e45
I20260619 16:54:03.723169 1314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:03.740520 1314 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:03.742439 1314 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:03.761190 1314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86: Generated new TSK 0
I20260619 16:54:03.762465 1314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:03.768829 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:33679
--builtin_ntp_servers=127.14.153.20:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:54:04.112391 1324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:04.112737 1324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:04.112898 1324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:04.120481 1324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:04.120630 1324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:04.120818 1324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:54:04.130141 1324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:33679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:04.131686 1324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:04.133497 1324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:04.146973 1329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:04.147001 1332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:04.146899 1330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:04.148053 1324 server_base.cc:1034] running on GCE node
I20260619 16:54:04.148694 1324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:04.149811 1324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:04.151095 1324 hybrid_clock.cc:648] HybridClock initialized: now 1781888044150997 us; error 48 us; skew 500 ppm
I20260619 16:54:04.151443 1324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:54:04.153898 1324 webserver.cc:460] Webserver started at http://127.14.153.1:40811/ using document root <none> and password file <none>
I20260619 16:54:04.154663 1324 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:04.154805 1324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:04.155200 1324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:04.158224 1324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "6fba8cc286d544d780459347c8d1e660"
format_stamp: "Formatted at 2026-06-19 16:54:04 on dist-test-slave-qb36"
server_key: "993876901339c80d7ec89de2c62c838b"
server_key_iv: "42f9a8e5d94621b049c30ee09871665b"
server_key_version: "encryptionkey@0"
I20260619 16:54:04.159137 1324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "6fba8cc286d544d780459347c8d1e660"
format_stamp: "Formatted at 2026-06-19 16:54:04 on dist-test-slave-qb36"
server_key: "993876901339c80d7ec89de2c62c838b"
server_key_iv: "42f9a8e5d94621b049c30ee09871665b"
server_key_version: "encryptionkey@0"
I20260619 16:54:04.164458 1324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 16:54:04.168056 1338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:04.169512 1324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 16:54:04.169682 1324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "6fba8cc286d544d780459347c8d1e660"
format_stamp: "Formatted at 2026-06-19 16:54:04 on dist-test-slave-qb36"
server_key: "993876901339c80d7ec89de2c62c838b"
server_key_iv: "42f9a8e5d94621b049c30ee09871665b"
server_key_version: "encryptionkey@0"
I20260619 16:54:04.169905 1324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:04.191493 1324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:04.192487 1324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:04.192811 1324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:04.193849 1324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:04.195477 1324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:04.195627 1324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:04.195758 1324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:04.195848 1324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:04.230822 1324 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:33909
I20260619 16:54:04.230914 1450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:33909 every 8 connection(s)
I20260619 16:54:04.232251 1324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:54:04.240891 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 1324
I20260619 16:54:04.241070 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:54:04.241600 14948 external_mini_cluster.cc:1428] Setting key b3125cba3913e22754e2b7c8ec06a9a1
I20260619 16:54:04.244503 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:33679
--builtin_ntp_servers=127.14.153.20:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:54:04.246227 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:04.235062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:55605 (local address 127.14.153.62:33679)
0619 16:54:04.235581 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:04.235587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:04.236898 (+ 1311us) server_negotiation.cc:407] Connection header received
0619 16:54:04.238725 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:04.238730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:04.238802 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:04.238905 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:04.240224 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:04.241307 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:04.243195 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:04.243760 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:04.243890 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:04.245239 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:04.245247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:04.245255 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:04.245281 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:04.245334 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:04.245353 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:04.245443 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:04.245664 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 16:54:04.245781 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":200,"threads_started":1}
I20260619 16:54:04.247193 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:04.235793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:33679 (local address 127.14.153.1:55605)
0619 16:54:04.236723 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:04.236767 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:04.238393 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:04.239295 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:04.239303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:04.239398 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:04.240020 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:04.240038 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:04.241465 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:04.241470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:04.242988 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:04.243000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:04.243973 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:04.243979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:04.244185 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:04.244893 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:54:04.244896 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:04.244937 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:54:04.245021 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:04.245446 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:04.245453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:04.245459 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:54:04.245596 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 16:54:04.245932 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":287,"threads_started":1}
I20260619 16:54:04.248750 1451 heartbeater.cc:347] Connected to a master server at 127.14.153.62:33679
I20260619 16:54:04.249090 1451 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:04.250042 1451 heartbeater.cc:511] Master 127.14.153.62:33679 requested a full tablet report, sending...
I20260619 16:54:04.252597 1265 ts_manager.cc:194] Registered new tserver with Master: 6fba8cc286d544d780459347c8d1e660 (127.14.153.1:33909)
I20260619 16:54:04.254861 1265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.1:55605
W20260619 16:54:04.577786 1455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:04.578107 1455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:04.578197 1455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:04.585589 1455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:04.585757 1455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:04.585871 1455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:54:04.595495 1455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:33679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:04.596837 1455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:04.598619 1455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:04.614162 1461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:04.614029 1463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:04.614053 1460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:04.615368 1455 server_base.cc:1034] running on GCE node
I20260619 16:54:04.616057 1455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:04.617085 1455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:04.618330 1455 hybrid_clock.cc:648] HybridClock initialized: now 1781888044618268 us; error 54 us; skew 500 ppm
I20260619 16:54:04.618683 1455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:54:04.621272 1455 webserver.cc:460] Webserver started at http://127.14.153.2:36061/ using document root <none> and password file <none>
I20260619 16:54:04.622044 1455 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:04.622191 1455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:04.622534 1455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:04.625577 1455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "ffec499d487e487bae3fc3a293edf769"
format_stamp: "Formatted at 2026-06-19 16:54:04 on dist-test-slave-qb36"
server_key: "186ce2b01c457bec11851f95f513cea2"
server_key_iv: "d950536d83116dde96231cd1fdf205b5"
server_key_version: "encryptionkey@0"
I20260619 16:54:04.626375 1455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "ffec499d487e487bae3fc3a293edf769"
format_stamp: "Formatted at 2026-06-19 16:54:04 on dist-test-slave-qb36"
server_key: "186ce2b01c457bec11851f95f513cea2"
server_key_iv: "d950536d83116dde96231cd1fdf205b5"
server_key_version: "encryptionkey@0"
I20260619 16:54:04.631788 1455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:54:04.635329 1469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:04.636623 1455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:54:04.636797 1455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "ffec499d487e487bae3fc3a293edf769"
format_stamp: "Formatted at 2026-06-19 16:54:04 on dist-test-slave-qb36"
server_key: "186ce2b01c457bec11851f95f513cea2"
server_key_iv: "d950536d83116dde96231cd1fdf205b5"
server_key_version: "encryptionkey@0"
I20260619 16:54:04.637007 1455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:04.649907 1455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:04.650751 1455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:04.651116 1455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:04.652129 1455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:04.653566 1455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:04.653652 1455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:04.653805 1455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:04.653878 1455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:04.688707 1455 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:45829
I20260619 16:54:04.688799 1581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:45829 every 8 connection(s)
I20260619 16:54:04.690127 1455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:54:04.691890 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 1455
I20260619 16:54:04.692095 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:54:04.692605 14948 external_mini_cluster.cc:1428] Setting key 3246c89a366f51c63baf35bfdf39e488
I20260619 16:54:04.695741 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:33679
--builtin_ntp_servers=127.14.153.20:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:54:04.704144 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:04.693125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:43835 (local address 127.14.153.62:33679)
0619 16:54:04.693327 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:04.693332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:04.694993 (+ 1661us) server_negotiation.cc:407] Connection header received
0619 16:54:04.696963 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:04.696972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:04.697071 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:04.697185 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:04.698275 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:04.699427 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:04.700995 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:04.701700 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:04.701845 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:04.703254 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:04.703262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:04.703269 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:04.703296 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:04.703341 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:04.703354 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:04.703425 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:04.703744 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:54:04.703849 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 16:54:04.705812 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:04.693741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:33679 (local address 127.14.153.2:43835)
0619 16:54:04.694860 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:04.694898 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:04.696652 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:04.697403 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:04.697411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:04.697508 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:04.698094 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:04.698111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:04.699570 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:04.699574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:04.700843 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:04.700855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:04.701904 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:04.701908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:04.702114 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:04.702869 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:54:04.702871 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:04.702913 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:54:04.702993 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:04.703451 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:04.703463 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:04.703472 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:54:04.703700 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 16:54:04.704158 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":295,"threads_started":1}
I20260619 16:54:04.707394 1582 heartbeater.cc:347] Connected to a master server at 127.14.153.62:33679
I20260619 16:54:04.707726 1582 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:04.708617 1582 heartbeater.cc:511] Master 127.14.153.62:33679 requested a full tablet report, sending...
I20260619 16:54:04.710253 1265 ts_manager.cc:194] Registered new tserver with Master: ffec499d487e487bae3fc3a293edf769 (127.14.153.2:45829)
I20260619 16:54:04.711184 1265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.2:43835
W20260619 16:54:05.030045 1585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:05.030339 1585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:05.030426 1585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:05.038208 1585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:05.038311 1585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:05.038470 1585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:54:05.048120 1585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:33679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:05.049484 1585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:05.051414 1585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:05.065383 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:05.065920 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:05.065946 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:05.066390 1585 server_base.cc:1034] running on GCE node
I20260619 16:54:05.066998 1585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:05.068053 1585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:05.069322 1585 hybrid_clock.cc:648] HybridClock initialized: now 1781888045069274 us; error 42 us; skew 500 ppm
I20260619 16:54:05.069706 1585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:54:05.072067 1585 webserver.cc:460] Webserver started at http://127.14.153.3:41093/ using document root <none> and password file <none>
I20260619 16:54:05.072793 1585 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:05.072923 1585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:05.073264 1585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:05.076431 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "708b407de4a7483386132fbe540638d7"
format_stamp: "Formatted at 2026-06-19 16:54:05 on dist-test-slave-qb36"
server_key: "9210cb9d85429069dc9d6ed778d327e5"
server_key_iv: "3732cd56a7318043a456bad397735ec2"
server_key_version: "encryptionkey@0"
I20260619 16:54:05.077263 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "708b407de4a7483386132fbe540638d7"
format_stamp: "Formatted at 2026-06-19 16:54:05 on dist-test-slave-qb36"
server_key: "9210cb9d85429069dc9d6ed778d327e5"
server_key_iv: "3732cd56a7318043a456bad397735ec2"
server_key_version: "encryptionkey@0"
I20260619 16:54:05.082475 1585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 16:54:05.086102 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:05.087746 1585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:05.087944 1585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "708b407de4a7483386132fbe540638d7"
format_stamp: "Formatted at 2026-06-19 16:54:05 on dist-test-slave-qb36"
server_key: "9210cb9d85429069dc9d6ed778d327e5"
server_key_iv: "3732cd56a7318043a456bad397735ec2"
server_key_version: "encryptionkey@0"
I20260619 16:54:05.088173 1585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:05.112291 1585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:05.113204 1585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:05.113547 1585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:05.114630 1585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:05.116207 1585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:05.116355 1585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:05.116492 1585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:05.116583 1585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:05.151064 1585 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:35691
I20260619 16:54:05.151118 1711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:35691 every 8 connection(s)
I20260619 16:54:05.152465 1585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:54:05.154670 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 1585
I20260619 16:54:05.154867 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:54:05.155426 14948 external_mini_cluster.cc:1428] Setting key b83ae1b7af68ba43f6b744fd52f90dcf
I20260619 16:54:05.165863 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.155496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:53251 (local address 127.14.153.62:33679)
0619 16:54:05.155726 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.155732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.157434 (+ 1702us) server_negotiation.cc:407] Connection header received
0619 16:54:05.159580 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.159586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.159680 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.159784 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:05.160836 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.161577 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.163148 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.163550 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.163691 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.165050 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:05.165055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:05.165059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:05.165077 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:05.165116 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:05.165128 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:05.165218 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.165442 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.165540 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 16:54:05.166958 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.156155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:33679 (local address 127.14.153.3:53251)
0619 16:54:05.157287 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.157331 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:05.159320 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:05.160012 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:05.160021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:05.160118 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:05.160678 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.160695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.161729 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.161740 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.162963 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.162974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.163780 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.163787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.164020 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.164722 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:54:05.164725 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:05.164765 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:54:05.164847 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:05.165231 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:05.165243 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:05.165257 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 16:54:05.165432 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 16:54:05.165843 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":319,"threads_started":1}
I20260619 16:54:05.168365 1712 heartbeater.cc:347] Connected to a master server at 127.14.153.62:33679
I20260619 16:54:05.168748 1712 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:05.169478 1712 heartbeater.cc:511] Master 127.14.153.62:33679 requested a full tablet report, sending...
I20260619 16:54:05.170892 1265 ts_manager.cc:194] Registered new tserver with Master: 708b407de4a7483386132fbe540638d7 (127.14.153.3:35691)
I20260619 16:54:05.171732 1265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.3:53251
I20260619 16:54:05.183364 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:54:05.194024 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.187857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.14.153.62:33679)
0619 16:54:05.188104 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.188110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.188418 (+ 308us) server_negotiation.cc:407] Connection header received
0619 16:54:05.188563 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.188567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.188642 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.188727 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:05.189300 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.189991 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.191518 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.191933 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.192079 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.193263 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:05.193268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:05.193271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:05.193289 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:05.193321 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:05.193333 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:05.193449 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.193622 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.193669 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:54:05.198197 1265 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:54:05.205413 1265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:54:05.231388 1724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.218323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:45829 (local address 127.0.0.1:52862)
0619 16:54:05.219228 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.219268 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:05.219451 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:05.220740 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:05.220751 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:05.220794 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:05.221384 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.221401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.224096 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.224105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.227363 (+ 3258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.227389 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.230378 (+ 2989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.230385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.230618 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.230670 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 16:54:05.230806 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 16:54:05.231013 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":397,"threads_started":1}
I20260619 16:54:05.232797 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.218529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.14.153.2:45829)
0619 16:54:05.219127 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.219136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.219651 (+ 515us) server_negotiation.cc:407] Connection header received
0619 16:54:05.219775 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.219789 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.220132 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.220369 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:05.221618 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.223851 (+ 2233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.227689 (+ 3838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.229817 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.229962 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.230230 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.232262 (+ 2032us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.232443 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":613,"threads_started":1}
I20260619 16:54:05.241446 1517 tablet_service.cc:1467] Processing CreateTablet for tablet 4aeea40db8f14f0b9fe17ebb1548e723 (DEFAULT_TABLE table=test-table [id=4cc4a6c182214210b65242cbd520ce38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:05.243136 1517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aeea40db8f14f0b9fe17ebb1548e723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:05.245916 1724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.219001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33909 (local address 127.0.0.1:46800)
0619 16:54:05.231954 (+ 12953us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.231976 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:05.232084 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:05.238223 (+ 6139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:05.238227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:05.238248 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:05.238521 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.238532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.240614 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.240627 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.242748 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.242774 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.244467 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.244473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.244681 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.244719 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:54:05.244822 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:54:05.245572 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12816,"mutex_wait_us":68,"thread_start_us":15784,"threads_started":1}
I20260619 16:54:05.245422 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.219588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.14.153.1:33909)
0619 16:54:05.237596 (+ 18008us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.237604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.237633 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:54:05.237741 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.237749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.238010 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.238201 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:05.238702 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.240402 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.242929 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.244296 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.244440 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.244719 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.244864 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.245037 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17830,"thread_start_us":17686,"threads_started":1}
I20260619 16:54:05.247107 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.219744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:35691 (local address 127.0.0.1:54644)
0619 16:54:05.235131 (+ 15387us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.235152 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:05.235272 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:05.237927 (+ 2655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:05.237931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:05.237950 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:05.238243 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.238253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.240536 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.240542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.243778 (+ 3236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.243794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.245939 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.245947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.246494 (+ 547us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.246532 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:54:05.246625 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:54:05.246761 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15212,"mutex_wait_us":79,"thread_start_us":17275,"threads_started":1}
I20260619 16:54:05.247237 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.220375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.14.153.3:35691)
0619 16:54:05.235698 (+ 15323us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.235710 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.235771 (+ 61us) server_negotiation.cc:407] Connection header received
0619 16:54:05.235904 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.235917 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.236269 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.236531 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:05.238508 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.240394 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.243998 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.245383 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.245530 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.245798 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.246694 (+ 896us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.246869 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15097,"thread_start_us":14941,"threads_started":1}
I20260619 16:54:05.250161 1386 tablet_service.cc:1467] Processing CreateTablet for tablet 4aeea40db8f14f0b9fe17ebb1548e723 (DEFAULT_TABLE table=test-table [id=4cc4a6c182214210b65242cbd520ce38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:05.251736 1386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aeea40db8f14f0b9fe17ebb1548e723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:05.253156 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 4aeea40db8f14f0b9fe17ebb1548e723 (DEFAULT_TABLE table=test-table [id=4cc4a6c182214210b65242cbd520ce38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:05.254479 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aeea40db8f14f0b9fe17ebb1548e723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:05.258227 1730 tablet_bootstrap.cc:492] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: Bootstrap starting.
I20260619 16:54:05.258200 1451 heartbeater.cc:503] Master 127.14.153.62:33679 was elected leader, sending a full tablet report...
I20260619 16:54:05.263536 1730 tablet_bootstrap.cc:654] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:05.264875 1730 log.cc:824] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:05.265962 1731 tablet_bootstrap.cc:492] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660: Bootstrap starting.
I20260619 16:54:05.268909 1733 tablet_bootstrap.cc:492] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: Bootstrap starting.
I20260619 16:54:05.269867 1731 tablet_bootstrap.cc:654] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:05.271202 1730 tablet_bootstrap.cc:492] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: No bootstrap required, opened a new log
I20260619 16:54:05.271190 1731 log.cc:824] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:05.271492 1730 ts_tablet_manager.cc:1397] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260619 16:54:05.272524 1733 tablet_bootstrap.cc:654] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:05.273835 1733 log.cc:824] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:05.274457 1731 tablet_bootstrap.cc:492] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660: No bootstrap required, opened a new log
I20260619 16:54:05.274755 1731 ts_tablet_manager.cc:1397] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 16:54:05.277027 1733 tablet_bootstrap.cc:492] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: No bootstrap required, opened a new log
I20260619 16:54:05.277292 1733 ts_tablet_manager.cc:1397] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 16:54:05.280468 1730 raft_consensus.cc:348] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } }
I20260619 16:54:05.280879 1730 raft_consensus.cc:374] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:05.280975 1730 raft_consensus.cc:729] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffec499d487e487bae3fc3a293edf769, State: Initialized, Role: FOLLOWER
I20260619 16:54:05.281594 1730 consensus_queue.cc:260] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } }
I20260619 16:54:05.285475 1582 heartbeater.cc:503] Master 127.14.153.62:33679 was elected leader, sending a full tablet report...
I20260619 16:54:05.286083 1730 ts_tablet_manager.cc:1428] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260619 16:54:05.286522 1731 raft_consensus.cc:348] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } }
I20260619 16:54:05.286934 1731 raft_consensus.cc:374] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:05.287103 1731 raft_consensus.cc:729] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fba8cc286d544d780459347c8d1e660, State: Initialized, Role: FOLLOWER
I20260619 16:54:05.287822 1733 raft_consensus.cc:348] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } }
I20260619 16:54:05.287914 1731 consensus_queue.cc:260] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } }
I20260619 16:54:05.288194 1733 raft_consensus.cc:374] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:05.288321 1733 raft_consensus.cc:729] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708b407de4a7483386132fbe540638d7, State: Initialized, Role: FOLLOWER
I20260619 16:54:05.289021 1733 consensus_queue.cc:260] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } }
I20260619 16:54:05.291563 1712 heartbeater.cc:503] Master 127.14.153.62:33679 was elected leader, sending a full tablet report...
I20260619 16:54:05.291687 1733 ts_tablet_manager.cc:1428] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260619 16:54:05.294342 1731 ts_tablet_manager.cc:1428] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
I20260619 16:54:05.363286 1737 raft_consensus.cc:482] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:54:05.363674 1737 raft_consensus.cc:504] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } }
I20260619 16:54:05.365494 1737 leader_election.cc:288] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffec499d487e487bae3fc3a293edf769 (127.14.153.2:45829), 708b407de4a7483386132fbe540638d7 (127.14.153.3:35691)
I20260619 16:54:05.372649 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.365864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:45829 (local address 127.14.153.1:47341)
0619 16:54:05.366431 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.366447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:05.366656 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:05.367143 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:05.367148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:05.367180 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:05.367593 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.367605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.368839 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.368844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.370504 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.370516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.371912 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.371919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.372101 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.372142 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:54:05.372256 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:54:05.372391 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":308,"threads_started":1}
I20260619 16:54:05.372721 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.365937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:47341 (local address 127.14.153.2:45829)
0619 16:54:05.366205 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.366211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.366579 (+ 368us) server_negotiation.cc:407] Connection header received
0619 16:54:05.366821 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.366830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.366943 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.367062 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:05.367772 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.368709 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.370752 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.371731 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.371850 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.371900 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.372321 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.372462 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:54:05.373234 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.366928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:51999 (local address 127.14.153.3:35691)
0619 16:54:05.367163 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.367169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.367362 (+ 193us) server_negotiation.cc:407] Connection header received
0619 16:54:05.367518 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.367526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.367605 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.367704 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:05.368361 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.369251 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.371229 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.372160 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.372312 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.372374 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.372872 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.373009 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:54:05.373257 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.366624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:35691 (local address 127.14.153.1:51999)
0619 16:54:05.367263 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.367277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:05.367391 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:05.367769 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:05.367773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:05.367807 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:05.368197 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.368210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.369397 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.369401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.371007 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:05.371022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.372377 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.372384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:05.372588 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.372628 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:54:05.372730 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:54:05.372856 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"spinlock_wait_cycles":3328,"thread_start_us":275,"threads_started":1}
I20260619 16:54:05.373888 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aeea40db8f14f0b9fe17ebb1548e723" candidate_uuid: "6fba8cc286d544d780459347c8d1e660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffec499d487e487bae3fc3a293edf769" is_pre_election: true
I20260619 16:54:05.374341 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aeea40db8f14f0b9fe17ebb1548e723" candidate_uuid: "6fba8cc286d544d780459347c8d1e660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "708b407de4a7483386132fbe540638d7" is_pre_election: true
I20260619 16:54:05.374473 1537 raft_consensus.cc:2454] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fba8cc286d544d780459347c8d1e660 in term 0.
I20260619 16:54:05.374905 1667 raft_consensus.cc:2454] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fba8cc286d544d780459347c8d1e660 in term 0.
I20260619 16:54:05.375186 1341 leader_election.cc:302] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fba8cc286d544d780459347c8d1e660, ffec499d487e487bae3fc3a293edf769; no voters:
I20260619 16:54:05.375576 1737 raft_consensus.cc:2781] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:54:05.375731 1737 raft_consensus.cc:482] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:54:05.375851 1737 raft_consensus.cc:3037] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:05.378003 1737 raft_consensus.cc:504] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } }
I20260619 16:54:05.378595 1737 leader_election.cc:288] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [CANDIDATE]: Term 1 election: Requested vote from peers ffec499d487e487bae3fc3a293edf769 (127.14.153.2:45829), 708b407de4a7483386132fbe540638d7 (127.14.153.3:35691)
I20260619 16:54:05.379117 1537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aeea40db8f14f0b9fe17ebb1548e723" candidate_uuid: "6fba8cc286d544d780459347c8d1e660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffec499d487e487bae3fc3a293edf769"
I20260619 16:54:05.379313 1537 raft_consensus.cc:3037] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:05.379288 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aeea40db8f14f0b9fe17ebb1548e723" candidate_uuid: "6fba8cc286d544d780459347c8d1e660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "708b407de4a7483386132fbe540638d7"
I20260619 16:54:05.379500 1667 raft_consensus.cc:3037] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:05.381542 1537 raft_consensus.cc:2454] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fba8cc286d544d780459347c8d1e660 in term 1.
I20260619 16:54:05.381714 1667 raft_consensus.cc:2454] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fba8cc286d544d780459347c8d1e660 in term 1.
I20260619 16:54:05.381953 1341 leader_election.cc:302] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fba8cc286d544d780459347c8d1e660, ffec499d487e487bae3fc3a293edf769; no voters:
I20260619 16:54:05.382323 1737 raft_consensus.cc:2781] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:05.382810 1737 raft_consensus.cc:686] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 1 LEADER]: Becoming Leader. State: Replica: 6fba8cc286d544d780459347c8d1e660, State: Running, Role: LEADER
I20260619 16:54:05.383353 1737 consensus_queue.cc:237] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } }
I20260619 16:54:05.388360 1264 catalog_manager.cc:5495] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fba8cc286d544d780459347c8d1e660 (127.14.153.1). New cstate: current_term: 1 leader_uuid: "6fba8cc286d544d780459347c8d1e660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fba8cc286d544d780459347c8d1e660" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 33909 } health_report { overall_health: HEALTHY } } }
W20260619 16:54:05.404327 1713 tablet.cc:2369] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:54:05.406296 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.400822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.14.153.1:33909)
0619 16:54:05.401016 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.401021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.401037 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:54:05.401121 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.401124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.401212 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.401338 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:05.401697 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.402396 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.403782 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.404365 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.404504 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.405098 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:05.405677 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:05.405817 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.405967 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.406036 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:54:05.414752 1537 raft_consensus.cc:1261] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [term 1 FOLLOWER]: Refusing update from remote peer 6fba8cc286d544d780459347c8d1e660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:05.414752 1667 raft_consensus.cc:1261] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [term 1 FOLLOWER]: Refusing update from remote peer 6fba8cc286d544d780459347c8d1e660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:05.415920 1741 consensus_queue.cc:1035] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffec499d487e487bae3fc3a293edf769" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:05.416332 1737 consensus_queue.cc:1035] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "708b407de4a7483386132fbe540638d7" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:05.428633 1744 mvcc.cc:205] Tried to move back new op lower bound from 7298613434008760320 to 7298613433893867520. Current Snapshot: MvccSnapshot[applied={T|T < 7298613434008760320}]
I20260619 16:54:05.444427 1264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:54:05.444751 1264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:54:05.448221 1264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6e3aad600c24dda88fdbad582a57f86: Sending DeleteTablet for 3 replicas of tablet 4aeea40db8f14f0b9fe17ebb1548e723
I20260619 16:54:05.449455 1386 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aeea40db8f14f0b9fe17ebb1548e723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:05 UTC) from {username='slave'} at 127.0.0.1:46800
I20260619 16:54:05.449447 1517 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aeea40db8f14f0b9fe17ebb1548e723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:05 UTC) from {username='slave'} at 127.0.0.1:52862
I20260619 16:54:05.449599 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aeea40db8f14f0b9fe17ebb1548e723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:05 UTC) from {username='slave'} at 127.0.0.1:54644
I20260619 16:54:05.451622 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 1324
I20260619 16:54:05.460350 1756 tablet_replica.cc:330] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: stopping tablet replica
I20260619 16:54:05.460955 1756 raft_consensus.cc:2229] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:54:05.461617 1756 raft_consensus.cc:2258] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:05.464932 1754 tablet_replica.cc:330] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: stopping tablet replica
I20260619 16:54:05.465391 1754 raft_consensus.cc:2229] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:54:05.465932 1754 raft_consensus.cc:2258] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:05.466176 1756 ts_tablet_manager.cc:1905] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:05.468246 1754 ts_tablet_manager.cc:1905] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:05.472115 1755 tablet_replica.cc:330] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660: stopping tablet replica
I20260619 16:54:05.472606 1755 raft_consensus.cc:2229] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:54:05.473522 1755 raft_consensus.cc:2258] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:05.475761 1754 ts_tablet_manager.cc:1918] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:05.475924 1754 log.cc:1196] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/4aeea40db8f14f0b9fe17ebb1548e723
I20260619 16:54:05.476470 1754 ts_tablet_manager.cc:1939] T 4aeea40db8f14f0b9fe17ebb1548e723 P ffec499d487e487bae3fc3a293edf769: Deleting consensus metadata
I20260619 16:54:05.477393 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.453194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.14.153.1:33909)
0619 16:54:05.453374 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.453380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.467731 (+ 14351us) server_negotiation.cc:407] Connection header received
0619 16:54:05.468322 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.468328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.468444 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.468529 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:05.469278 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.470580 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.472715 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.473472 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.473576 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.475336 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:05.475347 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:05.475365 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:05.476203 (+ 838us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:05.476261 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:05.476290 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:05.476374 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.476805 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.476910 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:54:05.477172 1755 ts_tablet_manager.cc:1905] T 4aeea40db8f14f0b9fe17ebb1548e723 P 6fba8cc286d544d780459347c8d1e660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:05.478410 1251 catalog_manager.cc:4841] TS ffec499d487e487bae3fc3a293edf769 (127.14.153.2:45829): tablet 4aeea40db8f14f0b9fe17ebb1548e723 (table test-table [id=4cc4a6c182214210b65242cbd520ce38]) successfully deleted
I20260619 16:54:05.478686 1446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:05.481757 1756 ts_tablet_manager.cc:1918] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:05.482003 1756 log.cc:1196] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/4aeea40db8f14f0b9fe17ebb1548e723
I20260619 16:54:05.482769 1756 ts_tablet_manager.cc:1939] T 4aeea40db8f14f0b9fe17ebb1548e723 P 708b407de4a7483386132fbe540638d7: Deleting consensus metadata
I20260619 16:54:05.484673 1251 catalog_manager.cc:4841] TS 708b407de4a7483386132fbe540638d7 (127.14.153.3:35691): tablet 4aeea40db8f14f0b9fe17ebb1548e723 (table test-table [id=4cc4a6c182214210b65242cbd520ce38]) successfully deleted
I20260619 16:54:05.576503 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 1324
W20260619 16:54:05.595048 1250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.153.1:33909 (error 108)
I20260619 16:54:05.595249 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 1455
I20260619 16:54:05.596932 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.596518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33909 (local address 127.0.0.1:46840)
0619 16:54:05.596684 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.596793 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 16:54:05.597147 1250 catalog_manager.cc:4584] TS 6fba8cc286d544d780459347c8d1e660 (127.14.153.1:33909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4aeea40db8f14f0b9fe17ebb1548e723: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33909: connect: Connection refused (error 111)
I20260619 16:54:05.602517 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.596187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.14.153.2:45829)
0619 16:54:05.596370 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.596377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.596397 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:54:05.596508 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.596513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.596602 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.596713 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:05.597180 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.598023 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.599495 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.599907 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.600010 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.601312 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:05.601320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:05.601334 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:05.601907 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:05.601953 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:05.601971 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:05.602057 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.602172 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.602262 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:54:05.603240 1577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:05.651243 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.650549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33909 (local address 127.0.0.1:46844)
0619 16:54:05.650853 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.650985 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 16:54:05.652400 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.652015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33909 (local address 127.0.0.1:46850)
0619 16:54:05.652181 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.652265 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:54:05.695884 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 1455
I20260619 16:54:05.713014 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.712406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33909 (local address 127.0.0.1:46858)
0619 16:54:05.712659 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.712765 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 16:54:05.713456 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 1585
I20260619 16:54:05.714133 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.713780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33909 (local address 127.0.0.1:46874)
0619 16:54:05.713931 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.713999 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:54:05.719919 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.714213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.14.153.3:35691)
0619 16:54:05.714384 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:05.714390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:05.714413 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:54:05.714570 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:05.714574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:05.714662 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:05.714767 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:05.715129 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.715857 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.717061 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:05.717438 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:05.717530 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:05.718582 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:05.718592 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:05.718606 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:05.719245 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:05.719292 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:05.719312 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:05.719405 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:05.719567 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 16:54:05.719697 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:54:05.720564 1707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:05.794974 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.794473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33909 (local address 127.0.0.1:46878)
0619 16:54:05.794688 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.794793 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 16:54:05.796118 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:05.795717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:33909 (local address 127.0.0.1:46886)
0619 16:54:05.795927 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:05.796009 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 16:54:05.814144 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 1585
I20260619 16:54:05.830741 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 1234
I20260619 16:54:05.831912 1295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:05.944258 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 1234
2026-06-19T16:54:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2803 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:06 dist-test-slave-qb36 krb5kdc[1765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:06 dist-test-slave-qb36 krb5kdc[1765](info): set up 2 sockets
Jun 19 16:54:06 dist-test-slave-qb36 krb5kdc[1765](info): commencing operation
krb5kdc: starting...
W20260619 16:54:07.051676 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:07 dist-test-slave-qb36 krb5kdc[1765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:07.310420 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:37177
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:39557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:37177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:07.641731 1778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:07.642019 1778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:07.642109 1778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:07.649279 1778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:07.649394 1778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:07.649441 1778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:07.649500 1778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:07.649545 1778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:07.659755 1778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:39557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:37177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:37177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:07.661154 1778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:07.662460 1778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:07.674790 1784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:07.674722 1786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:07.675380 1783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:07.676074 1778 server_base.cc:1034] running on GCE node
I20260619 16:54:07.677048 1778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:07.678671 1778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:07.679939 1778 hybrid_clock.cc:648] HybridClock initialized: now 1781888047679878 us; error 78 us; skew 500 ppm
I20260619 16:54:07.680356 1778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:07 dist-test-slave-qb36 krb5kdc[1765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888047, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:07.687181 1778 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:07.688938 1778 webserver.cc:460] Webserver started at http://127.14.153.62:44219/ using document root <none> and password file <none>
I20260619 16:54:07.689706 1778 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:07.689858 1778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:07.690182 1778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:07.693403 1778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "6b40f59f5b894f639af0b3a52cce6b50"
format_stamp: "Formatted at 2026-06-19 16:54:07 on dist-test-slave-qb36"
server_key: "85093ed67563f3f52b537b42a5ab80fc"
server_key_iv: "01a1e3a518f0dfc16f3bfd26a104517f"
server_key_version: "encryptionkey@0"
I20260619 16:54:07.694199 1778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "6b40f59f5b894f639af0b3a52cce6b50"
format_stamp: "Formatted at 2026-06-19 16:54:07 on dist-test-slave-qb36"
server_key: "85093ed67563f3f52b537b42a5ab80fc"
server_key_iv: "01a1e3a518f0dfc16f3bfd26a104517f"
server_key_version: "encryptionkey@0"
I20260619 16:54:07.699848 1778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 16:54:07.703708 1793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:07.705188 1778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:07.705390 1778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "6b40f59f5b894f639af0b3a52cce6b50"
format_stamp: "Formatted at 2026-06-19 16:54:07 on dist-test-slave-qb36"
server_key: "85093ed67563f3f52b537b42a5ab80fc"
server_key_iv: "01a1e3a518f0dfc16f3bfd26a104517f"
server_key_version: "encryptionkey@0"
I20260619 16:54:07.705608 1778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:07.719393 1778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:07.724088 1778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:07.724412 1778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:07.743654 1778 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:37177
I20260619 16:54:07.743688 1844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:37177 every 8 connection(s)
I20260619 16:54:07.745070 1778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:07.748983 1845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:07.755357 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 1778
I20260619 16:54:07.755569 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:07.755990 14948 external_mini_cluster.cc:1428] Setting key af2314fc5f49d9df017951688f81aad6
I20260619 16:54:07.762742 1845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50: Bootstrap starting.
I20260619 16:54:07.766999 1845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:07.768168 1845 log.cc:824] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:07.771672 1845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50: No bootstrap required, opened a new log
Jun 19 16:54:07 dist-test-slave-qb36 krb5kdc[1765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:07.785127 1845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b40f59f5b894f639af0b3a52cce6b50" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 37177 } }
I20260619 16:54:07.785625 1845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:07.785681 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:07.757942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.14.153.62:37177)
0619 16:54:07.758873 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:07.758891 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:07.758948 (+ 57us) server_negotiation.cc:407] Connection header received
0619 16:54:07.761666 (+ 2718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:07.761703 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:07.762313 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:07.762865 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:07.764428 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:07.766697 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:07.768317 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:07.769382 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:07.769531 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:07.774244 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:07.774256 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:07.774284 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:07.774343 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:07.779168 (+ 4825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:07.780334 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:07.780348 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:07.780360 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:07.780640 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:07.781210 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:07.781216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:07.781220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:07.782239 (+ 1019us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:07.782582 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:07.783416 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 16:54:07.784064 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":346,"threads_started":1}
I20260619 16:54:07.785809 1845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b40f59f5b894f639af0b3a52cce6b50, State: Initialized, Role: FOLLOWER
I20260619 16:54:07.786594 1845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b40f59f5b894f639af0b3a52cce6b50" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 37177 } }
I20260619 16:54:07.786978 1845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:07.787178 1845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:07.787405 1845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:07.791306 1845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b40f59f5b894f639af0b3a52cce6b50" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 37177 } }
I20260619 16:54:07.791878 1845 leader_election.cc:302] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b40f59f5b894f639af0b3a52cce6b50; no voters:
I20260619 16:54:07.792534 1845 leader_election.cc:288] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:07.792655 1850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:07.793459 1850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [term 1 LEADER]: Becoming Leader. State: Replica: 6b40f59f5b894f639af0b3a52cce6b50, State: Running, Role: LEADER
I20260619 16:54:07.794019 1850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b40f59f5b894f639af0b3a52cce6b50" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 37177 } }
I20260619 16:54:07.794941 1845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:07.796802 1852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b40f59f5b894f639af0b3a52cce6b50. Latest consensus state: current_term: 1 leader_uuid: "6b40f59f5b894f639af0b3a52cce6b50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b40f59f5b894f639af0b3a52cce6b50" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 37177 } } }
I20260619 16:54:07.797168 1852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:07.796871 1851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b40f59f5b894f639af0b3a52cce6b50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b40f59f5b894f639af0b3a52cce6b50" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 37177 } } }
I20260619 16:54:07.797283 1851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:07.797943 1855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:07.811010 1855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:07.830255 1855 catalog_manager.cc:1347] Generated new cluster ID: e6e7be4b38e64a2194fa587ed5146a05
I20260619 16:54:07.830396 1855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:07.852164 1855 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:07.853540 1855 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:07.862285 1855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50: Generated new TSK 0
I20260619 16:54:07.863256 1855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
Entry for principal kudu/127.14.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab.
I20260619 16:54:07.934903 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:37177
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:39557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:08.284582 1871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:08.284873 1871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:08.285038 1871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:08.292313 1871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:08.292428 1871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:08.292629 1871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:54:08.302897 1871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:39557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.1.keytab
--principal=kudu/127.14.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:37177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:08.304364 1871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:08.305728 1871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:08.319303 1876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:08.319367 1877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:08.319332 1879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:08.320523 1871 server_base.cc:1034] running on GCE node
I20260619 16:54:08.321168 1871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:08.322105 1871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:08.323335 1871 hybrid_clock.cc:648] HybridClock initialized: now 1781888048323291 us; error 52 us; skew 500 ppm
I20260619 16:54:08.323745 1871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:08 dist-test-slave-qb36 krb5kdc[1765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888048, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:08.327073 1871 init.cc:377] Logged in from keytab as kudu/127.14.153.1@KRBTEST.COM (short username kudu)
I20260619 16:54:08.328615 1871 webserver.cc:460] Webserver started at http://127.14.153.1:44773/ using document root <none> and password file <none>
I20260619 16:54:08.329356 1871 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:08.329506 1871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:08.329842 1871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:08.332888 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "e7412404c5a848ceafdd7c928fd88f7c"
format_stamp: "Formatted at 2026-06-19 16:54:08 on dist-test-slave-qb36"
server_key: "7caaa78847be499cdaa8623eb068cf00"
server_key_iv: "7404a5cbbdbba464fa134e64852b7e50"
server_key_version: "encryptionkey@0"
I20260619 16:54:08.333673 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "e7412404c5a848ceafdd7c928fd88f7c"
format_stamp: "Formatted at 2026-06-19 16:54:08 on dist-test-slave-qb36"
server_key: "7caaa78847be499cdaa8623eb068cf00"
server_key_iv: "7404a5cbbdbba464fa134e64852b7e50"
server_key_version: "encryptionkey@0"
I20260619 16:54:08.339277 1871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:54:08.342715 1886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:08.344105 1871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:08.344280 1871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "e7412404c5a848ceafdd7c928fd88f7c"
format_stamp: "Formatted at 2026-06-19 16:54:08 on dist-test-slave-qb36"
server_key: "7caaa78847be499cdaa8623eb068cf00"
server_key_iv: "7404a5cbbdbba464fa134e64852b7e50"
server_key_version: "encryptionkey@0"
I20260619 16:54:08.344489 1871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:08.370278 1871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:08.374785 1871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:08.375157 1871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:08.376169 1871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:08.377537 1871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:08.377650 1871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:08.377803 1871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:08.377882 1871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:08.411475 1871 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:35523
I20260619 16:54:08.411564 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:35523 every 8 connection(s)
I20260619 16:54:08.412842 1871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:54:08.415970 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 1871
I20260619 16:54:08.416157 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:54:08.416779 14948 external_mini_cluster.cc:1428] Setting key 56808da26d9463b6f08248149a42e52a
Jun 19 16:54:08 dist-test-slave-qb36 krb5kdc[1765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888048, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.1@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:08.437546 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:08.415505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:56947 (local address 127.14.153.62:37177)
0619 16:54:08.416034 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:08.416038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:08.417327 (+ 1289us) server_negotiation.cc:407] Connection header received
0619 16:54:08.419835 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:08.419839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:08.419917 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:08.420024 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:08.422266 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:08.423023 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:08.425645 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:08.426048 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:08.426153 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:08.430092 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:08.430101 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:08.430109 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:08.430157 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:08.434097 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:08.434848 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:08.434857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:08.434863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:08.435013 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:08.435458 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:08.435466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:08.435472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:08.435921 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:08.436067 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:08.436776 (+ 709us) server_negotiation.cc:299] Negotiation successful
0619 16:54:08.437037 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":308,"threads_started":1}
I20260619 16:54:08.438165 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:08.416333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:37177 (local address 127.14.153.1:56947)
0619 16:54:08.417164 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:08.417208 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:08.419478 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:08.420333 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:08.420348 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:08.421207 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:08.422081 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:08.422110 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:08.423270 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:08.423278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:08.425491 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:08.425509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:08.426289 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:08.426299 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:08.426643 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:08.427456 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:08.427493 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:08.429857 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:08.434246 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:08.434253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:08.434266 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:08.434723 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:08.435157 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:08.435160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:08.435163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:08.435339 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:08.436050 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:08.436056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:08.436384 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 16:54:08.436618 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 16:54:08.437015 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":329,"threads_started":1}
I20260619 16:54:08.439906 1999 heartbeater.cc:347] Connected to a master server at 127.14.153.62:37177
I20260619 16:54:08.440255 1999 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:08.441177 1999 heartbeater.cc:511] Master 127.14.153.62:37177 requested a full tablet report, sending...
I20260619 16:54:08.443651 1810 ts_manager.cc:194] Registered new tserver with Master: e7412404c5a848ceafdd7c928fd88f7c (127.14.153.1:35523)
WARNING: no policy specified for kudu/127.14.153.2@KRBTEST.COM; defaulting to no policy
I20260619 16:54:08.445218 1810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.1@KRBTEST.COM'} at 127.14.153.1:56947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
Entry for principal kudu/127.14.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab.
I20260619 16:54:08.476179 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:37177
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:39557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:08.811849 2005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:08.812143 2005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:08.812302 2005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:08.819798 2005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:08.819916 2005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:08.820097 2005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:54:08.830220 2005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:39557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.2.keytab
--principal=kudu/127.14.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:37177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:08.831636 2005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:08.832957 2005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:08.847164 2010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:08.847091 2013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:08.847136 2011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:08.848097 2005 server_base.cc:1034] running on GCE node
I20260619 16:54:08.848749 2005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:08.849769 2005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:08.850997 2005 hybrid_clock.cc:648] HybridClock initialized: now 1781888048850934 us; error 52 us; skew 500 ppm
I20260619 16:54:08.851374 2005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:08 dist-test-slave-qb36 krb5kdc[1765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888048, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:08.854596 2005 init.cc:377] Logged in from keytab as kudu/127.14.153.2@KRBTEST.COM (short username kudu)
I20260619 16:54:08.856124 2005 webserver.cc:460] Webserver started at http://127.14.153.2:33835/ using document root <none> and password file <none>
I20260619 16:54:08.856829 2005 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:08.856980 2005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:08.857303 2005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:08.860371 2005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "58a78436039346c6badafd1a5c7ae459"
format_stamp: "Formatted at 2026-06-19 16:54:08 on dist-test-slave-qb36"
server_key: "99c1120eca581ae8bf8830586b272d32"
server_key_iv: "47c75a91ed7c65afa071b8117d1d0983"
server_key_version: "encryptionkey@0"
I20260619 16:54:08.861171 2005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "58a78436039346c6badafd1a5c7ae459"
format_stamp: "Formatted at 2026-06-19 16:54:08 on dist-test-slave-qb36"
server_key: "99c1120eca581ae8bf8830586b272d32"
server_key_iv: "47c75a91ed7c65afa071b8117d1d0983"
server_key_version: "encryptionkey@0"
I20260619 16:54:08.866487 2005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 16:54:08.870132 2020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:08.871659 2005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:08.871840 2005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "58a78436039346c6badafd1a5c7ae459"
format_stamp: "Formatted at 2026-06-19 16:54:08 on dist-test-slave-qb36"
server_key: "99c1120eca581ae8bf8830586b272d32"
server_key_iv: "47c75a91ed7c65afa071b8117d1d0983"
server_key_version: "encryptionkey@0"
I20260619 16:54:08.872071 2005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:08.898994 2005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:08.903800 2005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:08.904131 2005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:08.905169 2005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:08.906569 2005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:08.906644 2005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:08.906790 2005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:08.906841 2005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:08.940869 2005 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:35497
I20260619 16:54:08.940948 2132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:35497 every 8 connection(s)
I20260619 16:54:08.942191 2005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:54:08.951074 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 2005
I20260619 16:54:08.951323 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:54:08.951783 14948 external_mini_cluster.cc:1428] Setting key b3eb3824e07230c295a21a72410d0718
Jun 19 16:54:08 dist-test-slave-qb36 krb5kdc[1765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888048, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.2@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:08.965648 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:08.944714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:44207 (local address 127.14.153.62:37177)
0619 16:54:08.945160 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:08.945167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:08.946215 (+ 1048us) server_negotiation.cc:407] Connection header received
0619 16:54:08.948201 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:08.948206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:08.948283 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:08.948404 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:08.949938 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:08.950907 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:08.952796 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:08.953427 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:08.953556 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:08.958073 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:08.958083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:08.958091 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:08.958143 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:08.962290 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:08.963090 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:08.963097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:08.963102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:08.963268 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:08.963683 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:08.963691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:08.963698 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:08.964073 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:08.964188 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:08.965166 (+ 978us) server_negotiation.cc:299] Negotiation successful
0619 16:54:08.965342 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":201,"threads_started":1}
I20260619 16:54:08.967231 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:08.945342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:37177 (local address 127.14.153.2:44207)
0619 16:54:08.946050 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:08.946093 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:08.947842 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:08.948685 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:08.948693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:08.949206 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:08.949770 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:08.949788 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:08.951112 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:08.951117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:08.952569 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:08.952593 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:08.953756 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:08.953768 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:08.954091 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:08.954881 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:08.954916 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:08.957815 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:08.962443 (+ 4628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:08.962451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:08.962466 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:08.962930 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:08.963375 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:08.963378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:08.963381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:08.963561 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:08.964200 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:08.964211 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:08.964709 (+ 498us) client_negotiation.cc:769] Sending connection context
0619 16:54:08.965059 (+ 350us) client_negotiation.cc:240] Negotiation successful
0619 16:54:08.965679 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":300,"threads_started":1}
I20260619 16:54:08.968902 2133 heartbeater.cc:347] Connected to a master server at 127.14.153.62:37177
I20260619 16:54:08.969283 2133 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:08.970108 2133 heartbeater.cc:511] Master 127.14.153.62:37177 requested a full tablet report, sending...
I20260619 16:54:08.971805 1810 ts_manager.cc:194] Registered new tserver with Master: 58a78436039346c6badafd1a5c7ae459 (127.14.153.2:35497)
I20260619 16:54:08.972674 1810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.2@KRBTEST.COM'} at 127.14.153.2:44207
WARNING: no policy specified for kudu/127.14.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
Entry for principal kudu/127.14.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab.
I20260619 16:54:09.012600 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:37177
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:39557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:09.350942 2139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:09.351271 2139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:09.351373 2139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:09.359488 2139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:09.359611 2139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:09.359724 2139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:54:09.369711 2139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:39557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.3.keytab
--principal=kudu/127.14.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:37177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:09.370978 2139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:09.372310 2139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:09.386008 2144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:09.385983 2145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:09.386029 2147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:09.392290 2139 server_base.cc:1034] running on GCE node
I20260619 16:54:09.393019 2139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:09.394063 2139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:09.395296 2139 hybrid_clock.cc:648] HybridClock initialized: now 1781888049395230 us; error 66 us; skew 500 ppm
I20260619 16:54:09.395643 2139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:09 dist-test-slave-qb36 krb5kdc[1765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888049, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:09.399015 2139 init.cc:377] Logged in from keytab as kudu/127.14.153.3@KRBTEST.COM (short username kudu)
I20260619 16:54:09.400784 2139 webserver.cc:460] Webserver started at http://127.14.153.3:44125/ using document root <none> and password file <none>
I20260619 16:54:09.401510 2139 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:09.401679 2139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:09.402001 2139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:09.405076 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "37732186fabe4519bffe5f1da93c2d9c"
format_stamp: "Formatted at 2026-06-19 16:54:09 on dist-test-slave-qb36"
server_key: "a7e39a63e7c76f48c0709bf2cecc11b5"
server_key_iv: "3cfa12de78dd671e22d19890557ce025"
server_key_version: "encryptionkey@0"
I20260619 16:54:09.405862 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "37732186fabe4519bffe5f1da93c2d9c"
format_stamp: "Formatted at 2026-06-19 16:54:09 on dist-test-slave-qb36"
server_key: "a7e39a63e7c76f48c0709bf2cecc11b5"
server_key_iv: "3cfa12de78dd671e22d19890557ce025"
server_key_version: "encryptionkey@0"
I20260619 16:54:09.411404 2139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 16:54:09.415071 2154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:09.416577 2139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:09.416771 2139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "37732186fabe4519bffe5f1da93c2d9c"
format_stamp: "Formatted at 2026-06-19 16:54:09 on dist-test-slave-qb36"
server_key: "a7e39a63e7c76f48c0709bf2cecc11b5"
server_key_iv: "3cfa12de78dd671e22d19890557ce025"
server_key_version: "encryptionkey@0"
I20260619 16:54:09.417001 2139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:09.434741 2139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:09.439308 2139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:09.439610 2139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:09.440622 2139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:09.442029 2139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:09.442137 2139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:09.442283 2139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:09.442354 2139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:09.448594 1999 heartbeater.cc:503] Master 127.14.153.62:37177 was elected leader, sending a full tablet report...
I20260619 16:54:09.476320 2139 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:45597
I20260619 16:54:09.476405 2266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:45597 every 8 connection(s)
I20260619 16:54:09.477727 2139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:54:09.488095 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 2139
I20260619 16:54:09.488308 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:54:09.488888 14948 external_mini_cluster.cc:1428] Setting key 8dc9b049cded4562ea5ab1d8e4e63b9f
Jun 19 16:54:09 dist-test-slave-qb36 krb5kdc[1765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888049, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.3@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:09.498167 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.480350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:37351 (local address 127.14.153.62:37177)
0619 16:54:09.480808 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.480813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.481837 (+ 1024us) server_negotiation.cc:407] Connection header received
0619 16:54:09.483700 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.483705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.483790 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.483890 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:09.485369 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.486157 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.487837 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.488478 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.488615 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.492414 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:09.492419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:09.492422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:09.492454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:09.495237 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:09.496013 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:09.496024 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:09.496026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:09.496184 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:09.496655 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:09.496659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:09.496661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:09.496932 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:09.497070 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.497704 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.497875 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":225,"threads_started":1}
I20260619 16:54:09.499182 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.480951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:37177 (local address 127.14.153.3:37351)
0619 16:54:09.481665 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:09.481709 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:09.483437 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:09.484097 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:09.484105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:09.484606 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:09.485198 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.485215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.486298 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.486301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.487696 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.487708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.488704 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.488710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.488957 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.489760 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:09.489786 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:09.492218 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:09.495350 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:09.495356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:09.495369 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:09.495892 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:09.496350 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:09.496354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:09.496357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:09.496537 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:09.497046 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:09.497051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:09.497364 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 16:54:09.497587 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 16:54:09.498002 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":276,"threads_started":1}
I20260619 16:54:09.500638 2267 heartbeater.cc:347] Connected to a master server at 127.14.153.62:37177
I20260619 16:54:09.500974 2267 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:09.501672 2267 heartbeater.cc:511] Master 127.14.153.62:37177 requested a full tablet report, sending...
I20260619 16:54:09.503393 1809 ts_manager.cc:194] Registered new tserver with Master: 37732186fabe4519bffe5f1da93c2d9c (127.14.153.3:45597)
I20260619 16:54:09.504354 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:54:09.504536 1809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.153.3@KRBTEST.COM'} at 127.14.153.3:37351
I20260619 16:54:09.520874 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.508633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.14.153.62:37177)
0619 16:54:09.508789 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.508793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.509309 (+ 516us) server_negotiation.cc:407] Connection header received
0619 16:54:09.509453 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.509457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.509518 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.509596 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:09.510990 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.511788 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.513002 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.513390 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.513491 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.515770 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:09.515778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:09.515782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:09.515822 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:09.518408 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:09.518953 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:09.518956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:09.518959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:09.519140 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:09.519435 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:09.519438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:09.519440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:09.519765 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:09.519961 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.520339 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.520574 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:54:09.524760 1809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:54:09.531733 1809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:54:09.561342 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.544256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:45597 (local address 127.0.0.1:39766)
0619 16:54:09.545426 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:09.545474 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:09.545676 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:09.548414 (+ 2738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:09.548425 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:09.548467 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:09.549039 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.549057 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.553840 (+ 4783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.553848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.558276 (+ 4428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.558290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.560243 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.560249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.560456 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.560516 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 16:54:09.560641 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 16:54:09.560890 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":7716,"threads_started":1}
I20260619 16:54:09.562243 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.544836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.14.153.3:45597)
0619 16:54:09.546368 (+ 1532us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.546381 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.546423 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:54:09.546530 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.546537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.546791 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.549283 (+ 2492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:09.549580 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.553646 (+ 4066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.558467 (+ 4821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.560014 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.560165 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.561615 (+ 1450us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.561698 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.561874 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":1172,"threads_started":1}
I20260619 16:54:09.565025 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.544276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:35523 (local address 127.0.0.1:48786)
0619 16:54:09.552301 (+ 8025us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:09.552328 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:09.552471 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:09.558718 (+ 6247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:09.558725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:09.558756 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:09.559185 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.559199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.560716 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.560720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.562692 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.562705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.564237 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.564245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.564417 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.564462 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 16:54:09.564571 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:54:09.564700 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7861,"thread_start_us":669,"threads_started":1}
I20260619 16:54:09.565083 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.544301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.14.153.1:35523)
0619 16:54:09.558107 (+ 13806us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.558116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.558145 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:54:09.558243 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.558249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.558500 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.558702 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:09.559354 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.560538 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.562847 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.564008 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.564169 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.564400 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.564623 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.564786 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13642,"thread_start_us":13490,"threads_started":1}
I20260619 16:54:09.566679 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.546563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:35497 (local address 127.0.0.1:43404)
0619 16:54:09.556887 (+ 10324us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:09.556913 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:09.557075 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:09.558398 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:09.558403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:09.558428 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:09.558885 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.558902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.560806 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.560810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.564127 (+ 3317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.564146 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.565943 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.565950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.566115 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.566148 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 16:54:09.566239 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:54:09.566360 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10132,"thread_start_us":11453,"threads_started":1}
I20260619 16:54:09.566978 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.546768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.14.153.2:35497)
0619 16:54:09.557742 (+ 10974us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.557755 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.557794 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:54:09.557896 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.557906 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.558195 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.558417 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:09.559128 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.560619 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.564343 (+ 3724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.565723 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.565872 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.566051 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.566463 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.566632 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10739,"thread_start_us":10583,"threads_started":1}
I20260619 16:54:09.566370 2202 tablet_service.cc:1467] Processing CreateTablet for tablet 4623699feea74b2a8d30ab70b2bc43e2 (DEFAULT_TABLE table=test-table [id=5b1f4c0fe48144b4aa63cf035d4aace1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:09.568984 2202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4623699feea74b2a8d30ab70b2bc43e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:09.568964 1934 tablet_service.cc:1467] Processing CreateTablet for tablet 4623699feea74b2a8d30ab70b2bc43e2 (DEFAULT_TABLE table=test-table [id=5b1f4c0fe48144b4aa63cf035d4aace1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:09.570048 2068 tablet_service.cc:1467] Processing CreateTablet for tablet 4623699feea74b2a8d30ab70b2bc43e2 (DEFAULT_TABLE table=test-table [id=5b1f4c0fe48144b4aa63cf035d4aace1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:09.570422 1934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4623699feea74b2a8d30ab70b2bc43e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:09.571120 2068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4623699feea74b2a8d30ab70b2bc43e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:09.582634 2286 tablet_bootstrap.cc:492] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: Bootstrap starting.
I20260619 16:54:09.584581 2287 tablet_bootstrap.cc:492] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: Bootstrap starting.
I20260619 16:54:09.586359 2288 tablet_bootstrap.cc:492] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: Bootstrap starting.
I20260619 16:54:09.588637 2286 tablet_bootstrap.cc:654] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:09.588639 2287 tablet_bootstrap.cc:654] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:09.589610 2288 tablet_bootstrap.cc:654] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:09.590008 2286 log.cc:824] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:09.590008 2287 log.cc:824] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:09.590760 2288 log.cc:824] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:09.593412 2288 tablet_bootstrap.cc:492] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: No bootstrap required, opened a new log
I20260619 16:54:09.593473 2287 tablet_bootstrap.cc:492] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: No bootstrap required, opened a new log
I20260619 16:54:09.593641 2286 tablet_bootstrap.cc:492] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: No bootstrap required, opened a new log
I20260619 16:54:09.593760 2287 ts_tablet_manager.cc:1397] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 16:54:09.593757 2288 ts_tablet_manager.cc:1397] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 16:54:09.593892 2286 ts_tablet_manager.cc:1397] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260619 16:54:09.601802 2288 raft_consensus.cc:348] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } }
I20260619 16:54:09.602249 2288 raft_consensus.cc:374] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:09.602376 2288 raft_consensus.cc:729] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7412404c5a848ceafdd7c928fd88f7c, State: Initialized, Role: FOLLOWER
I20260619 16:54:09.603118 2288 consensus_queue.cc:260] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } }
I20260619 16:54:09.605113 2288 ts_tablet_manager.cc:1428] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 16:54:09.605137 2286 raft_consensus.cc:348] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } }
I20260619 16:54:09.605139 2287 raft_consensus.cc:348] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } }
I20260619 16:54:09.605614 2287 raft_consensus.cc:374] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:09.605623 2286 raft_consensus.cc:374] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:09.605722 2286 raft_consensus.cc:729] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a78436039346c6badafd1a5c7ae459, State: Initialized, Role: FOLLOWER
I20260619 16:54:09.605816 2287 raft_consensus.cc:729] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37732186fabe4519bffe5f1da93c2d9c, State: Initialized, Role: FOLLOWER
I20260619 16:54:09.606396 2286 consensus_queue.cc:260] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } }
I20260619 16:54:09.606534 2287 consensus_queue.cc:260] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } }
I20260619 16:54:09.607613 2133 heartbeater.cc:503] Master 127.14.153.62:37177 was elected leader, sending a full tablet report...
I20260619 16:54:09.608345 2286 ts_tablet_manager.cc:1428] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260619 16:54:09.609803 2267 heartbeater.cc:503] Master 127.14.153.62:37177 was elected leader, sending a full tablet report...
I20260619 16:54:09.610658 2287 ts_tablet_manager.cc:1428] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260619 16:54:09.627826 2293 raft_consensus.cc:482] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:54:09.628247 2293 raft_consensus.cc:504] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } }
I20260619 16:54:09.630205 2293 leader_election.cc:288] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7412404c5a848ceafdd7c928fd88f7c (127.14.153.1:35523), 37732186fabe4519bffe5f1da93c2d9c (127.14.153.3:45597)
I20260619 16:54:09.636871 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.631421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:59851 (local address 127.14.153.1:35523)
0619 16:54:09.631599 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.631604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.631620 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:54:09.631679 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.631683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.631737 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.631848 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:09.632490 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.633201 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.635006 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.635885 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.636074 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.636159 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.636485 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.636616 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:54:09.636895 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.630859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:34519 (local address 127.14.153.3:45597)
0619 16:54:09.631127 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.631136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.631794 (+ 658us) server_negotiation.cc:407] Connection header received
0619 16:54:09.631942 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.631947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.632017 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.632120 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:09.632679 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.633422 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.635152 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.635927 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.636119 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.636196 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.636512 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.636639 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:54:09.637019 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.630803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:45597 (local address 127.14.153.2:34519)
0619 16:54:09.631691 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:09.631704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:09.631831 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:09.632133 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:09.632138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:09.632160 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:09.632528 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.632536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.633559 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.633564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.634966 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.634978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.636153 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.636160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.636353 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.636376 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:54:09.636477 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:54:09.636613 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":5,"thread_start_us":1157,"threads_started":1}
I20260619 16:54:09.637300 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.630385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:35523 (local address 127.14.153.2:59851)
0619 16:54:09.631154 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:09.631174 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:09.631300 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:09.631868 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:09.631873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:09.631899 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:09.632338 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.632347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.633326 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.633332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.634848 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:09.634860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.636126 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.636135 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:09.636320 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.636351 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:54:09.636456 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:54:09.636591 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":24,"thread_start_us":372,"threads_started":1}
I20260619 16:54:09.637989 2222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4623699feea74b2a8d30ab70b2bc43e2" candidate_uuid: "58a78436039346c6badafd1a5c7ae459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37732186fabe4519bffe5f1da93c2d9c" is_pre_election: true
I20260619 16:54:09.638298 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4623699feea74b2a8d30ab70b2bc43e2" candidate_uuid: "58a78436039346c6badafd1a5c7ae459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7412404c5a848ceafdd7c928fd88f7c" is_pre_election: true
I20260619 16:54:09.638491 2222 raft_consensus.cc:2454] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58a78436039346c6badafd1a5c7ae459 in term 0.
I20260619 16:54:09.638785 1954 raft_consensus.cc:2454] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58a78436039346c6badafd1a5c7ae459 in term 0.
I20260619 16:54:09.639242 2024 leader_election.cc:302] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37732186fabe4519bffe5f1da93c2d9c, 58a78436039346c6badafd1a5c7ae459; no voters:
I20260619 16:54:09.639590 2293 raft_consensus.cc:2781] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:54:09.639740 2293 raft_consensus.cc:482] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:54:09.639845 2293 raft_consensus.cc:3037] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:09.642087 2293 raft_consensus.cc:504] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } }
I20260619 16:54:09.642680 2293 leader_election.cc:288] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [CANDIDATE]: Term 1 election: Requested vote from peers e7412404c5a848ceafdd7c928fd88f7c (127.14.153.1:35523), 37732186fabe4519bffe5f1da93c2d9c (127.14.153.3:45597)
I20260619 16:54:09.643178 2222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4623699feea74b2a8d30ab70b2bc43e2" candidate_uuid: "58a78436039346c6badafd1a5c7ae459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37732186fabe4519bffe5f1da93c2d9c"
I20260619 16:54:09.643178 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4623699feea74b2a8d30ab70b2bc43e2" candidate_uuid: "58a78436039346c6badafd1a5c7ae459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7412404c5a848ceafdd7c928fd88f7c"
I20260619 16:54:09.643381 1954 raft_consensus.cc:3037] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:09.643368 2222 raft_consensus.cc:3037] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:09.646013 2222 raft_consensus.cc:2454] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58a78436039346c6badafd1a5c7ae459 in term 1.
I20260619 16:54:09.646030 1954 raft_consensus.cc:2454] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58a78436039346c6badafd1a5c7ae459 in term 1.
I20260619 16:54:09.646420 2024 leader_election.cc:302] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37732186fabe4519bffe5f1da93c2d9c, 58a78436039346c6badafd1a5c7ae459; no voters:
I20260619 16:54:09.646724 2293 raft_consensus.cc:2781] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:09.647233 2293 raft_consensus.cc:686] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 1 LEADER]: Becoming Leader. State: Replica: 58a78436039346c6badafd1a5c7ae459, State: Running, Role: LEADER
I20260619 16:54:09.647696 2293 consensus_queue.cc:237] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } }
I20260619 16:54:09.652603 1808 catalog_manager.cc:5495] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58a78436039346c6badafd1a5c7ae459 (127.14.153.2). New cstate: current_term: 1 leader_uuid: "58a78436039346c6badafd1a5c7ae459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58a78436039346c6badafd1a5c7ae459" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 35497 } health_report { overall_health: HEALTHY } } }
W20260619 16:54:09.665295 2000 tablet.cc:2369] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:54:09.681638 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.676088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.14.153.2:35497)
0619 16:54:09.676411 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.676415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.676435 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:54:09.676528 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.676533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.676610 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.676809 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:09.677246 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.678000 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.679443 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.679858 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.679995 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.680479 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:09.681003 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:09.681165 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.681262 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.681350 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:54:09.690738 1954 raft_consensus.cc:1261] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [term 1 FOLLOWER]: Refusing update from remote peer 58a78436039346c6badafd1a5c7ae459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:09.690737 2222 raft_consensus.cc:1261] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [term 1 FOLLOWER]: Refusing update from remote peer 58a78436039346c6badafd1a5c7ae459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:09.691944 2297 consensus_queue.cc:1035] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7412404c5a848ceafdd7c928fd88f7c" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:09.692358 2293 consensus_queue.cc:1035] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37732186fabe4519bffe5f1da93c2d9c" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 45597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:54:09.694202 2134 tablet.cc:2369] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:54:09.710305 2301 mvcc.cc:205] Tried to move back new op lower bound from 7298613451520802816 to 7298613451361034240. Current Snapshot: MvccSnapshot[applied={T|T < 7298613451520802816}]
I20260619 16:54:09.710800 2302 mvcc.cc:205] Tried to move back new op lower bound from 7298613451520802816 to 7298613451361034240. Current Snapshot: MvccSnapshot[applied={T|T < 7298613451520802816}]
I20260619 16:54:09.720629 1808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:54:09.720932 1808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:54:09.724617 1808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b40f59f5b894f639af0b3a52cce6b50: Sending DeleteTablet for 3 replicas of tablet 4623699feea74b2a8d30ab70b2bc43e2
I20260619 16:54:09.725831 2202 tablet_service.cc:1514] Processing DeleteTablet for tablet 4623699feea74b2a8d30ab70b2bc43e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:09 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:39766
I20260619 16:54:09.725931 2068 tablet_service.cc:1514] Processing DeleteTablet for tablet 4623699feea74b2a8d30ab70b2bc43e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:09 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:43404
I20260619 16:54:09.726585 1934 tablet_service.cc:1514] Processing DeleteTablet for tablet 4623699feea74b2a8d30ab70b2bc43e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:09 UTC) from {username='kudu', principal='kudu/127.14.153.62@KRBTEST.COM'} at 127.0.0.1:48786
I20260619 16:54:09.727455 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 1871
I20260619 16:54:09.737635 2311 tablet_replica.cc:330] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: stopping tablet replica
I20260619 16:54:09.738462 2311 raft_consensus.cc:2229] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:54:09.739645 2311 raft_consensus.cc:2258] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:09.740765 2310 tablet_replica.cc:330] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: stopping tablet replica
I20260619 16:54:09.741293 2309 tablet_replica.cc:330] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: stopping tablet replica
I20260619 16:54:09.741258 2310 raft_consensus.cc:2229] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:54:09.741668 2309 raft_consensus.cc:2229] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:54:09.741923 2310 raft_consensus.cc:2258] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:09.742188 2309 raft_consensus.cc:2258] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:09.744300 2311 ts_tablet_manager.cc:1905] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:09.744334 2310 ts_tablet_manager.cc:1905] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:09.748591 2309 ts_tablet_manager.cc:1905] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:09.751515 2311 ts_tablet_manager.cc:1918] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:09.751748 2311 log.cc:1196] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/4623699feea74b2a8d30ab70b2bc43e2
I20260619 16:54:09.752380 2311 ts_tablet_manager.cc:1939] T 4623699feea74b2a8d30ab70b2bc43e2 P e7412404c5a848ceafdd7c928fd88f7c: Deleting consensus metadata
I20260619 16:54:09.754648 1795 catalog_manager.cc:4841] TS e7412404c5a848ceafdd7c928fd88f7c (127.14.153.1:35523): tablet 4623699feea74b2a8d30ab70b2bc43e2 (table test-table [id=5b1f4c0fe48144b4aa63cf035d4aace1]) successfully deleted
I20260619 16:54:09.755482 2309 ts_tablet_manager.cc:1918] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:09.755717 2309 log.cc:1196] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/4623699feea74b2a8d30ab70b2bc43e2
I20260619 16:54:09.756273 2309 ts_tablet_manager.cc:1939] T 4623699feea74b2a8d30ab70b2bc43e2 P 37732186fabe4519bffe5f1da93c2d9c: Deleting consensus metadata
I20260619 16:54:09.758167 1797 catalog_manager.cc:4841] TS 37732186fabe4519bffe5f1da93c2d9c (127.14.153.3:45597): tablet 4623699feea74b2a8d30ab70b2bc43e2 (table test-table [id=5b1f4c0fe48144b4aa63cf035d4aace1]) successfully deleted
I20260619 16:54:09.759363 2310 ts_tablet_manager.cc:1918] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:09.759593 2310 log.cc:1196] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/4623699feea74b2a8d30ab70b2bc43e2
I20260619 16:54:09.760246 2310 ts_tablet_manager.cc:1939] T 4623699feea74b2a8d30ab70b2bc43e2 P 58a78436039346c6badafd1a5c7ae459: Deleting consensus metadata
Jun 19 16:54:09 dist-test-slave-qb36 krb5kdc[1765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.1@KRBTEST.COM
I20260619 16:54:09.762100 1795 catalog_manager.cc:4841] TS 58a78436039346c6badafd1a5c7ae459 (127.14.153.2:35497): tablet 4623699feea74b2a8d30ab70b2bc43e2 (table test-table [id=5b1f4c0fe48144b4aa63cf035d4aace1]) successfully deleted
I20260619 16:54:09.769151 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.730265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.14.153.1:35523)
0619 16:54:09.730443 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.730451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.743123 (+ 12672us) server_negotiation.cc:407] Connection header received
0619 16:54:09.744111 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.744117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.744211 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.744279 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:09.747533 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.748692 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.754590 (+ 5898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.755303 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.755399 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.763135 (+ 7736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:09.763151 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:09.763166 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:09.763210 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:09.766201 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:09.766948 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:09.766956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:09.766958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:09.767148 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:09.767543 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:09.767548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:09.767552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:09.767922 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:09.768243 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.768541 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.768766 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:54:09.769829 1994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:09.863138 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 1871
I20260619 16:54:09.882162 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 2005
Jun 19 16:54:09 dist-test-slave-qb36 krb5kdc[1765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.2@KRBTEST.COM
I20260619 16:54:09.898252 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:09.883012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.14.153.2:35497)
0619 16:54:09.883269 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:09.883275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:09.883299 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:54:09.883413 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:09.883419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:09.883510 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:09.883634 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:09.884878 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.885710 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.886975 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:09.887562 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:09.887739 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:09.891971 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:09.891987 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:09.892003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:09.892049 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:09.895261 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:09.896038 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:09.896085 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:09.896089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:09.896259 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:09.896632 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:09.896638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:09.896642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:09.897021 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:09.897358 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:09.897625 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 16:54:09.897851 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:54:09.898940 2128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:10.014107 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 2005
I20260619 16:54:10.033597 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 2139
Jun 19 16:54:10 dist-test-slave-qb36 krb5kdc[1765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.3@KRBTEST.COM
I20260619 16:54:10.048717 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:10.034337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.14.153.3:45597)
0619 16:54:10.034502 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:10.034507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:10.034525 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:54:10.034665 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:10.034669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:10.034731 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:10.034824 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:10.036167 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:10.036877 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:10.038134 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:10.038511 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:10.038611 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:10.042944 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:10.042959 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:10.042976 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:10.043010 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:10.046155 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:10.046770 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:10.046779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:10.046782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:10.046900 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:10.047346 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:10.047350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:10.047353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:10.047733 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:10.047970 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:10.048251 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:54:10.048418 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:54:10.049394 2262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:10.140491 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 2139
I20260619 16:54:10.156939 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 1778
I20260619 16:54:10.158154 1840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:10.269820 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 1778
2026-06-19T16:54:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4324 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T16:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:10Z Disabled control of system clock
I20260619 16:54:10.348608 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42279
--webserver_interface=127.14.153.62
--webserver_port=0
--builtin_ntp_servers=127.14.153.20:45299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:54:10.687914 2320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:10.688225 2320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:10.688308 2320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:10.695961 2320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:10.696090 2320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:10.696130 2320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:10.696179 2320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:10.696220 2320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:10.705637 2320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:45299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:42279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:10.706928 2320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:10.708717 2320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:10.720538 2326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:10.720626 2325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:10.720595 2328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:10.721335 2320 server_base.cc:1034] running on GCE node
I20260619 16:54:10.722136 2320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:10.723686 2320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:10.724931 2320 hybrid_clock.cc:648] HybridClock initialized: now 1781888050724890 us; error 35 us; skew 500 ppm
I20260619 16:54:10.725319 2320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:54:10.727837 2320 webserver.cc:460] Webserver started at http://127.14.153.62:35953/ using document root <none> and password file <none>
I20260619 16:54:10.728564 2320 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:10.728713 2320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:10.729043 2320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:10.732112 2320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "94d78e9c2735456595d0b857291f0077"
format_stamp: "Formatted at 2026-06-19 16:54:10 on dist-test-slave-qb36"
server_key: "697dd268afcbf6717467f68dad3ad9f1"
server_key_iv: "f47a5c85ba04481cc4a1d685573711c4"
server_key_version: "encryptionkey@0"
I20260619 16:54:10.732959 2320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "94d78e9c2735456595d0b857291f0077"
format_stamp: "Formatted at 2026-06-19 16:54:10 on dist-test-slave-qb36"
server_key: "697dd268afcbf6717467f68dad3ad9f1"
server_key_iv: "f47a5c85ba04481cc4a1d685573711c4"
server_key_version: "encryptionkey@0"
I20260619 16:54:10.738282 2320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:54:10.741945 2334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:10.743407 2320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:54:10.743623 2320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "94d78e9c2735456595d0b857291f0077"
format_stamp: "Formatted at 2026-06-19 16:54:10 on dist-test-slave-qb36"
server_key: "697dd268afcbf6717467f68dad3ad9f1"
server_key_iv: "f47a5c85ba04481cc4a1d685573711c4"
server_key_version: "encryptionkey@0"
I20260619 16:54:10.743860 2320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:10.764846 2320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:10.765829 2320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:10.766153 2320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:10.785389 2320 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:42279
I20260619 16:54:10.785413 2385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:42279 every 8 connection(s)
I20260619 16:54:10.786832 2320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:10.791095 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:10.794204 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 2320
I20260619 16:54:10.794394 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:10.794997 14948 external_mini_cluster.cc:1428] Setting key 4357f84285e1dc5b5e4ddca78710f3db
I20260619 16:54:10.803341 2386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077: Bootstrap starting.
I20260619 16:54:10.806493 2386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:10.807729 2386 log.cc:824] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:10.809250 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:10.797114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.14.153.62:42279)
0619 16:54:10.798125 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:10.798139 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:10.798179 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:54:10.799917 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:10.799941 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:10.800436 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:10.800861 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:10.801500 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:10.802750 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:10.804225 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:10.804964 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:10.805186 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:10.806360 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:10.806368 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:10.806383 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:10.807014 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:10.807168 (+ 154us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:10.807191 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:10.807315 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:10.807545 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 16:54:10.807910 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":342,"threads_started":1}
I20260619 16:54:10.811127 2386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077: No bootstrap required, opened a new log
I20260619 16:54:10.818660 2386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d78e9c2735456595d0b857291f0077" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42279 } }
I20260619 16:54:10.819109 2386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:10.819221 2386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d78e9c2735456595d0b857291f0077, State: Initialized, Role: FOLLOWER
I20260619 16:54:10.819849 2386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d78e9c2735456595d0b857291f0077" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42279 } }
I20260619 16:54:10.820084 2386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:10.820219 2386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:10.820389 2386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:10.822830 2386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d78e9c2735456595d0b857291f0077" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42279 } }
I20260619 16:54:10.823352 2386 leader_election.cc:302] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94d78e9c2735456595d0b857291f0077; no voters:
I20260619 16:54:10.823910 2386 leader_election.cc:288] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:10.824090 2391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:10.824877 2391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [term 1 LEADER]: Becoming Leader. State: Replica: 94d78e9c2735456595d0b857291f0077, State: Running, Role: LEADER
I20260619 16:54:10.825423 2391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d78e9c2735456595d0b857291f0077" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42279 } }
I20260619 16:54:10.826161 2386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:10.828665 2393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94d78e9c2735456595d0b857291f0077. Latest consensus state: current_term: 1 leader_uuid: "94d78e9c2735456595d0b857291f0077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d78e9c2735456595d0b857291f0077" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42279 } } }
I20260619 16:54:10.828970 2392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94d78e9c2735456595d0b857291f0077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d78e9c2735456595d0b857291f0077" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 42279 } } }
I20260619 16:54:10.829253 2393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:10.829332 2392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:10.830098 2400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:10.836582 2400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:10.845607 2400 catalog_manager.cc:1347] Generated new cluster ID: 8df8bec950934f119876011079a5b402
I20260619 16:54:10.845728 2400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:10.871790 2400 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:10.873375 2400 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:10.882468 2400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077: Generated new TSK 0
I20260619 16:54:10.883432 2400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:10.892212 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.1:0
--local_ip_for_outbound_sockets=127.14.153.1
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42279
--builtin_ntp_servers=127.14.153.20:45299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:54:11.230189 2410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:11.230574 2410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:11.230731 2410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:11.238226 2410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:11.238325 2410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:11.238502 2410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.1
I20260619 16:54:11.248211 2410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:45299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.153.1
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.1
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:11.249579 2410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:11.251391 2410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:11.265513 2415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:11.265429 2416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:11.265504 2418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:11.266322 2410 server_base.cc:1034] running on GCE node
I20260619 16:54:11.267019 2410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:11.268044 2410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:11.269294 2410 hybrid_clock.cc:648] HybridClock initialized: now 1781888051269247 us; error 42 us; skew 500 ppm
I20260619 16:54:11.269639 2410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:54:11.271945 2410 webserver.cc:460] Webserver started at http://127.14.153.1:41639/ using document root <none> and password file <none>
I20260619 16:54:11.272650 2410 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:11.272782 2410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:11.273100 2410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:11.276264 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/data/instance:
uuid: "73fba737befc4e44838982e2097777c7"
format_stamp: "Formatted at 2026-06-19 16:54:11 on dist-test-slave-qb36"
server_key: "1455f214520f4d0eaa4c007b798effc1"
server_key_iv: "e95affd4fd8ad9b9596a266b1e54dfac"
server_key_version: "encryptionkey@0"
I20260619 16:54:11.277060 2410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance:
uuid: "73fba737befc4e44838982e2097777c7"
format_stamp: "Formatted at 2026-06-19 16:54:11 on dist-test-slave-qb36"
server_key: "1455f214520f4d0eaa4c007b798effc1"
server_key_iv: "e95affd4fd8ad9b9596a266b1e54dfac"
server_key_version: "encryptionkey@0"
I20260619 16:54:11.282238 2410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:54:11.285760 2424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:11.287210 2410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:11.287392 2410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/wal
uuid: "73fba737befc4e44838982e2097777c7"
format_stamp: "Formatted at 2026-06-19 16:54:11 on dist-test-slave-qb36"
server_key: "1455f214520f4d0eaa4c007b798effc1"
server_key_iv: "e95affd4fd8ad9b9596a266b1e54dfac"
server_key_version: "encryptionkey@0"
I20260619 16:54:11.287590 2410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:11.309213 2410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:11.310082 2410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:11.310410 2410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:11.311393 2410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:11.312868 2410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:11.312988 2410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:11.313124 2410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:11.313213 2410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:11.348212 2410 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.1:46845
I20260619 16:54:11.348273 2536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.1:46845 every 8 connection(s)
I20260619 16:54:11.349587 2410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/data/info.pb
I20260619 16:54:11.357441 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 2410
I20260619 16:54:11.357645 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/wal/instance
I20260619 16:54:11.358156 14948 external_mini_cluster.cc:1428] Setting key 3e7fd83e7825672480662a5153a4d5eb
I20260619 16:54:11.361050 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.2:0
--local_ip_for_outbound_sockets=127.14.153.2
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42279
--builtin_ntp_servers=127.14.153.20:45299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:54:11.363955 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:11.352227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.1:51043 (local address 127.14.153.62:42279)
0619 16:54:11.352719 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:11.352725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:11.354014 (+ 1289us) server_negotiation.cc:407] Connection header received
0619 16:54:11.356258 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:11.356263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:11.356342 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:11.356441 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:11.357910 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:11.358847 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:11.360811 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:11.361233 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:11.361355 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:11.362942 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:11.362951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:11.362959 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:11.362985 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:11.363072 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:11.363092 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:11.363192 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:11.363395 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 16:54:11.363507 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":229,"threads_started":1}
I20260619 16:54:11.364823 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:11.352869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42279 (local address 127.14.153.1:51043)
0619 16:54:11.353846 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:11.353884 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:11.355982 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:11.356721 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:11.356730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:11.356824 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:11.357719 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:11.357749 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:11.359000 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:11.359006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:11.360652 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:11.360665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:11.361593 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:11.361597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:11.361819 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:11.362553 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:54:11.362556 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:11.362605 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:54:11.362731 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:11.363185 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:11.363193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:11.363199 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:54:11.363349 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 16:54:11.363701 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":282,"threads_started":1}
I20260619 16:54:11.366439 2537 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42279
I20260619 16:54:11.366788 2537 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:11.367671 2537 heartbeater.cc:511] Master 127.14.153.62:42279 requested a full tablet report, sending...
I20260619 16:54:11.370141 2351 ts_manager.cc:194] Registered new tserver with Master: 73fba737befc4e44838982e2097777c7 (127.14.153.1:46845)
I20260619 16:54:11.374279 2351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.1:51043
W20260619 16:54:11.716868 2541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:11.717172 2541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:11.717264 2541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:11.724690 2541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:11.724805 2541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:11.724915 2541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.2
I20260619 16:54:11.734388 2541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:45299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.153.2
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.2
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:11.735785 2541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:11.737524 2541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:11.751076 2546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:11.751045 2547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:11.750975 2549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:11.752120 2541 server_base.cc:1034] running on GCE node
I20260619 16:54:11.752704 2541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:11.753598 2541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:11.754796 2541 hybrid_clock.cc:648] HybridClock initialized: now 1781888051754730 us; error 54 us; skew 500 ppm
I20260619 16:54:11.755191 2541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:54:11.757452 2541 webserver.cc:460] Webserver started at http://127.14.153.2:33915/ using document root <none> and password file <none>
I20260619 16:54:11.758236 2541 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:11.758354 2541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:11.758687 2541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:11.761819 2541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/data/instance:
uuid: "94f4a49f47484c4d8fa4e588861ed76c"
format_stamp: "Formatted at 2026-06-19 16:54:11 on dist-test-slave-qb36"
server_key: "e7e6a5937ee36fcc43d4d4a1240ef6cf"
server_key_iv: "ac3d525d856f6424fbffd4d3ca2a447d"
server_key_version: "encryptionkey@0"
I20260619 16:54:11.762706 2541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance:
uuid: "94f4a49f47484c4d8fa4e588861ed76c"
format_stamp: "Formatted at 2026-06-19 16:54:11 on dist-test-slave-qb36"
server_key: "e7e6a5937ee36fcc43d4d4a1240ef6cf"
server_key_iv: "ac3d525d856f6424fbffd4d3ca2a447d"
server_key_version: "encryptionkey@0"
I20260619 16:54:11.768098 2541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:54:11.771661 2555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:11.773033 2541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:54:11.773217 2541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/wal
uuid: "94f4a49f47484c4d8fa4e588861ed76c"
format_stamp: "Formatted at 2026-06-19 16:54:11 on dist-test-slave-qb36"
server_key: "e7e6a5937ee36fcc43d4d4a1240ef6cf"
server_key_iv: "ac3d525d856f6424fbffd4d3ca2a447d"
server_key_version: "encryptionkey@0"
I20260619 16:54:11.773424 2541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:11.804663 2541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:11.805531 2541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:11.805846 2541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:11.806803 2541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:11.808360 2541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:11.808446 2541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:11.808597 2541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:11.808674 2541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:11.841645 2541 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.2:38379
I20260619 16:54:11.841733 2667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.2:38379 every 8 connection(s)
I20260619 16:54:11.842965 2541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/data/info.pb
I20260619 16:54:11.846341 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 2541
I20260619 16:54:11.846508 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/wal/instance
I20260619 16:54:11.846992 14948 external_mini_cluster.cc:1428] Setting key cdcc8fb954c945e669fefe8b0e24dce5
I20260619 16:54:11.849753 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.153.3:0
--local_ip_for_outbound_sockets=127.14.153.3
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42279
--builtin_ntp_servers=127.14.153.20:45299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:54:11.857265 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:11.845683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:44393 (local address 127.14.153.62:42279)
0619 16:54:11.845861 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:11.845866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:11.847719 (+ 1853us) server_negotiation.cc:407] Connection header received
0619 16:54:11.849725 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:11.849731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:11.849813 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:11.849933 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:11.851220 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:11.852387 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:11.853985 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:11.854720 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:11.854870 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:11.856298 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:11.856307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:11.856314 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:11.856341 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:11.856381 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:11.856400 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:11.856499 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:11.856742 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 16:54:11.856858 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:54:11.858409 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:11.846327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42279 (local address 127.14.153.2:44393)
0619 16:54:11.847527 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:11.847590 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:11.849459 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:11.850282 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:11.850290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:11.850387 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:11.850980 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:11.850997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:11.852551 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:11.852555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:11.853834 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:11.853846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:11.854936 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:11.854941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:11.855239 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:11.855979 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:54:11.855982 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:11.856022 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:54:11.856109 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:11.856500 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:11.856508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:11.856513 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:54:11.856680 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 16:54:11.856983 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":409,"threads_started":1}
I20260619 16:54:11.859932 2668 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42279
I20260619 16:54:11.860260 2668 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:11.861016 2668 heartbeater.cc:511] Master 127.14.153.62:42279 requested a full tablet report, sending...
I20260619 16:54:11.862452 2351 ts_manager.cc:194] Registered new tserver with Master: 94f4a49f47484c4d8fa4e588861ed76c (127.14.153.2:38379)
I20260619 16:54:11.863363 2351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.2:44393
W20260619 16:54:12.182595 2671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:12.182965 2671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:12.183153 2671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:12.190568 2671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:12.190734 2671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:12.190927 2671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.153.3
I20260619 16:54:12.201092 2671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:45299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.153.3
--webserver_port=0
--tserver_master_addrs=127.14.153.62:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.153.3
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:12.202457 2671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:12.204339 2671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:12.218513 2676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:12.218537 2677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:12.218618 2679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:12.219359 2671 server_base.cc:1034] running on GCE node
I20260619 16:54:12.220001 2671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:12.221050 2671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:12.222313 2671 hybrid_clock.cc:648] HybridClock initialized: now 1781888052222242 us; error 65 us; skew 500 ppm
I20260619 16:54:12.222684 2671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 16:54:12.225162 2671 webserver.cc:460] Webserver started at http://127.14.153.3:34113/ using document root <none> and password file <none>
I20260619 16:54:12.225929 2671 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:12.226063 2671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:12.226429 2671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:12.229655 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/data/instance:
uuid: "7675e981c580478e9a85553fce6b1687"
format_stamp: "Formatted at 2026-06-19 16:54:12 on dist-test-slave-qb36"
server_key: "ec1a6c1437343de595ca5c221d2174c5"
server_key_iv: "d447de0717246c958e207e933ef08688"
server_key_version: "encryptionkey@0"
I20260619 16:54:12.230548 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance:
uuid: "7675e981c580478e9a85553fce6b1687"
format_stamp: "Formatted at 2026-06-19 16:54:12 on dist-test-slave-qb36"
server_key: "ec1a6c1437343de595ca5c221d2174c5"
server_key_iv: "d447de0717246c958e207e933ef08688"
server_key_version: "encryptionkey@0"
I20260619 16:54:12.236112 2671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 16:54:12.239701 2685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:12.241148 2671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:12.241339 2671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/wal
uuid: "7675e981c580478e9a85553fce6b1687"
format_stamp: "Formatted at 2026-06-19 16:54:12 on dist-test-slave-qb36"
server_key: "ec1a6c1437343de595ca5c221d2174c5"
server_key_iv: "d447de0717246c958e207e933ef08688"
server_key_version: "encryptionkey@0"
I20260619 16:54:12.241583 2671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:12.265120 2671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:12.266078 2671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:12.266400 2671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:12.267436 2671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:54:12.268937 2671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:54:12.269060 2671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:12.269212 2671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:54:12.269320 2671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:12.304811 2671 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.3:35049
I20260619 16:54:12.304913 2797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.3:35049 every 8 connection(s)
I20260619 16:54:12.306177 2671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/data/info.pb
I20260619 16:54:12.306666 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 2671
I20260619 16:54:12.306852 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/wal/instance
I20260619 16:54:12.307489 14948 external_mini_cluster.cc:1428] Setting key c630463e1d1e17cfbfe07608370b5eef
I20260619 16:54:12.319240 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.309002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.3:55669 (local address 127.14.153.62:42279)
0619 16:54:12.309191 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.309198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.310972 (+ 1774us) server_negotiation.cc:407] Connection header received
0619 16:54:12.312836 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.312843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.312939 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.313046 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:12.314108 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.314856 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.316477 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.316882 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.317010 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.318379 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:12.318387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:12.318390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:12.318407 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:12.318438 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:12.318449 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:12.318538 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.318777 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.318878 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:54:12.320276 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.309658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:42279 (local address 127.14.153.3:55669)
0619 16:54:12.310807 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.310855 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:12.312567 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:12.313283 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:12.313291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:12.313379 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:12.313944 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.313959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.314997 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.315003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.316331 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.316342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.317106 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.317113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.317368 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.318039 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:54:12.318042 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:12.318080 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:54:12.318177 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:12.318553 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:12.318564 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:12.318578 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 16:54:12.318765 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 16:54:12.319187 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":346,"threads_started":1}
I20260619 16:54:12.321879 2798 heartbeater.cc:347] Connected to a master server at 127.14.153.62:42279
I20260619 16:54:12.322230 2798 heartbeater.cc:464] Registering TS with master...
I20260619 16:54:12.323233 2798 heartbeater.cc:511] Master 127.14.153.62:42279 requested a full tablet report, sending...
I20260619 16:54:12.324693 2351 ts_manager.cc:194] Registered new tserver with Master: 7675e981c580478e9a85553fce6b1687 (127.14.153.3:35049)
I20260619 16:54:12.325521 2351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.153.3:55669
I20260619 16:54:12.335803 14948 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:54:12.345991 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.339938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.14.153.62:42279)
0619 16:54:12.340148 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.340152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.340613 (+ 461us) server_negotiation.cc:407] Connection header received
0619 16:54:12.340762 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.340766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.340846 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.340939 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:12.341403 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.342133 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.343491 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.343891 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.343989 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.345129 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:12.345134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:12.345137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:12.345155 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:12.345188 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:12.345200 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:12.345330 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.345549 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.345705 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:54:12.350234 2351 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:54:12.357029 2351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:54:12.378310 2537 heartbeater.cc:503] Master 127.14.153.62:42279 was elected leader, sending a full tablet report...
I20260619 16:54:12.382051 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.370278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58334 (local address 127.14.153.3:35049)
0619 16:54:12.372084 (+ 1806us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.372096 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.372138 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:54:12.372255 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.372265 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.372545 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.373190 (+ 645us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:12.374614 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.376123 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.379325 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.380794 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.380934 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.381216 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.381466 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.381707 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1601,"thread_start_us":1457,"threads_started":1}
I20260619 16:54:12.382071 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.368978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:35049 (local address 127.0.0.1:58334)
0619 16:54:12.369914 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.369955 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:12.370131 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:12.372844 (+ 2713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:12.372855 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:12.372937 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:12.374230 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.374244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.376262 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.376267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.379093 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.379115 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.380985 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.380992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.381201 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.381255 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:54:12.381403 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 16:54:12.381626 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":174,"spinlock_wait_cycles":12672,"thread_start_us":469,"threads_started":1}
I20260619 16:54:12.382735 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.369754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.2:38379 (local address 127.0.0.1:42232)
0619 16:54:12.374645 (+ 4891us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.374663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:12.374785 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:12.375397 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:12.375402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:12.375427 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:12.375789 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.375800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.377309 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.377317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.380060 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.380081 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.381958 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.381964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.382174 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.382210 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:54:12.382293 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:54:12.382429 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4774,"thread_start_us":4628,"threads_started":1}
I20260619 16:54:12.383535 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.371238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.14.153.2:38379)
0619 16:54:12.373832 (+ 2594us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.373840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.374779 (+ 939us) server_negotiation.cc:407] Connection header received
0619 16:54:12.374924 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.374932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.375205 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.375391 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:12.375940 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.377132 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.380271 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.381480 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.381598 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.381825 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.383013 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.383194 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2436,"thread_start_us":2011,"threads_started":1}
I20260619 16:54:12.383548 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.370643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.14.153.1:46845)
0619 16:54:12.371764 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.371777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.371813 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:54:12.371935 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.371946 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.372280 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.372505 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:12.374465 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.375978 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.379318 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.380794 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.380934 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.381214 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.382970 (+ 1756us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.383189 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":722,"threads_started":1}
I20260619 16:54:12.384227 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.369149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:46845 (local address 127.0.0.1:54004)
0619 16:54:12.370616 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.370632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:12.370744 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:12.372842 (+ 2098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:12.372853 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:12.372917 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:12.374231 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.374244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.376094 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.376101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.379097 (+ 2996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.379115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.382562 (+ 3447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.382569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.382739 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.382779 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:54:12.383742 (+ 963us) client_negotiation.cc:240] Negotiation successful
0619 16:54:12.383888 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"mutex_wait_us":259,"spinlock_wait_cycles":3328,"thread_start_us":1210,"threads_started":1}
I20260619 16:54:12.386094 2733 tablet_service.cc:1467] Processing CreateTablet for tablet 30c7a14f10d0473380ee92d939da41cb (DEFAULT_TABLE table=test-table [id=e93369c6e8694d16afc80b93597016c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:12.387655 2733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c7a14f10d0473380ee92d939da41cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:12.387322 2603 tablet_service.cc:1467] Processing CreateTablet for tablet 30c7a14f10d0473380ee92d939da41cb (DEFAULT_TABLE table=test-table [id=e93369c6e8694d16afc80b93597016c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:12.387871 2472 tablet_service.cc:1467] Processing CreateTablet for tablet 30c7a14f10d0473380ee92d939da41cb (DEFAULT_TABLE table=test-table [id=e93369c6e8694d16afc80b93597016c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:54:12.388753 2603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c7a14f10d0473380ee92d939da41cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:12.389297 2472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c7a14f10d0473380ee92d939da41cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:12.399091 2816 tablet_bootstrap.cc:492] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: Bootstrap starting.
I20260619 16:54:12.403240 2816 tablet_bootstrap.cc:654] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:12.403805 2817 tablet_bootstrap.cc:492] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: Bootstrap starting.
I20260619 16:54:12.404603 2818 tablet_bootstrap.cc:492] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: Bootstrap starting.
I20260619 16:54:12.404716 2816 log.cc:824] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:12.408349 2818 tablet_bootstrap.cc:654] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:12.408589 2816 tablet_bootstrap.cc:492] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: No bootstrap required, opened a new log
I20260619 16:54:12.408852 2817 tablet_bootstrap.cc:654] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:12.408913 2816 ts_tablet_manager.cc:1397] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260619 16:54:12.410226 2818 log.cc:824] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:12.410547 2817 log.cc:824] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:12.414728 2818 tablet_bootstrap.cc:492] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: No bootstrap required, opened a new log
I20260619 16:54:12.415130 2818 ts_tablet_manager.cc:1397] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 16:54:12.417346 2816 raft_consensus.cc:348] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } }
I20260619 16:54:12.417755 2816 raft_consensus.cc:374] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:12.417796 2817 tablet_bootstrap.cc:492] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: No bootstrap required, opened a new log
I20260619 16:54:12.417869 2816 raft_consensus.cc:729] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7675e981c580478e9a85553fce6b1687, State: Initialized, Role: FOLLOWER
I20260619 16:54:12.418131 2817 ts_tablet_manager.cc:1397] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260619 16:54:12.418538 2816 consensus_queue.cc:260] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } }
I20260619 16:54:12.419718 2798 heartbeater.cc:503] Master 127.14.153.62:42279 was elected leader, sending a full tablet report...
I20260619 16:54:12.420452 2816 ts_tablet_manager.cc:1428] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260619 16:54:12.426165 2818 raft_consensus.cc:348] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } }
I20260619 16:54:12.426761 2818 raft_consensus.cc:374] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:12.426891 2818 raft_consensus.cc:729] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fba737befc4e44838982e2097777c7, State: Initialized, Role: FOLLOWER
I20260619 16:54:12.427565 2818 consensus_queue.cc:260] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } }
I20260619 16:54:12.427731 2817 raft_consensus.cc:348] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } }
I20260619 16:54:12.428154 2817 raft_consensus.cc:374] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:12.428272 2817 raft_consensus.cc:729] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f4a49f47484c4d8fa4e588861ed76c, State: Initialized, Role: FOLLOWER
I20260619 16:54:12.428855 2817 consensus_queue.cc:260] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } }
I20260619 16:54:12.429533 2818 ts_tablet_manager.cc:1428] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 16:54:12.429929 2668 heartbeater.cc:503] Master 127.14.153.62:42279 was elected leader, sending a full tablet report...
I20260619 16:54:12.430567 2817 ts_tablet_manager.cc:1428] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260619 16:54:12.435096 2824 raft_consensus.cc:482] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:54:12.435307 2824 raft_consensus.cc:504] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } }
I20260619 16:54:12.436753 2824 leader_election.cc:288] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7675e981c580478e9a85553fce6b1687 (127.14.153.3:35049), 73fba737befc4e44838982e2097777c7 (127.14.153.1:46845)
I20260619 16:54:12.446202 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.437304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:54761 (local address 127.14.153.1:46845)
0619 16:54:12.437524 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.437532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.438102 (+ 570us) server_negotiation.cc:407] Connection header received
0619 16:54:12.438308 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.438315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.438407 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.438566 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:12.439433 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.440548 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.443743 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.445040 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.445215 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.445287 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.445813 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.445951 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:54:12.446202 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.437285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.153.2:45401 (local address 127.14.153.3:35049)
0619 16:54:12.437530 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.437550 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.438135 (+ 585us) server_negotiation.cc:407] Connection header received
0619 16:54:12.438365 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.438372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.438450 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.438563 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:54:12.439433 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.440548 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.443743 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.445040 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.445215 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.445287 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.445839 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.445962 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:54:12.446336 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.437043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:46845 (local address 127.14.153.2:54761)
0619 16:54:12.437923 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.437950 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:12.438133 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:12.438547 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:12.438552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:12.438578 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:12.439244 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.439265 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.440705 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.440710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.443477 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.443493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.445303 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.445309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.445601 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.445642 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:54:12.445736 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:54:12.445969 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":306,"thread_start_us":672,"threads_started":1}
I20260619 16:54:12.446342 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.437185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.3:35049 (local address 127.14.153.2:45401)
0619 16:54:12.438041 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.438055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:12.438228 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:12.438565 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:12.438569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:12.438585 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:54:12.439282 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.439290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.440704 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.440710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.443484 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:12.443493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.445296 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.445303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:12.445602 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.445640 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:54:12.445736 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:54:12.445886 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":299,"thread_start_us":336,"threads_started":1}
I20260619 16:54:12.447579 2753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7a14f10d0473380ee92d939da41cb" candidate_uuid: "94f4a49f47484c4d8fa4e588861ed76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7675e981c580478e9a85553fce6b1687" is_pre_election: true
I20260619 16:54:12.447854 2492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7a14f10d0473380ee92d939da41cb" candidate_uuid: "94f4a49f47484c4d8fa4e588861ed76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73fba737befc4e44838982e2097777c7" is_pre_election: true
I20260619 16:54:12.448130 2753 raft_consensus.cc:2454] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f4a49f47484c4d8fa4e588861ed76c in term 0.
I20260619 16:54:12.448426 2492 raft_consensus.cc:2454] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f4a49f47484c4d8fa4e588861ed76c in term 0.
I20260619 16:54:12.448952 2556 leader_election.cc:302] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7675e981c580478e9a85553fce6b1687, 94f4a49f47484c4d8fa4e588861ed76c; no voters:
I20260619 16:54:12.449358 2824 raft_consensus.cc:2781] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:54:12.449501 2824 raft_consensus.cc:482] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:54:12.449602 2824 raft_consensus.cc:3037] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:12.451877 2824 raft_consensus.cc:504] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } }
I20260619 16:54:12.452509 2824 leader_election.cc:288] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [CANDIDATE]: Term 1 election: Requested vote from peers 7675e981c580478e9a85553fce6b1687 (127.14.153.3:35049), 73fba737befc4e44838982e2097777c7 (127.14.153.1:46845)
I20260619 16:54:12.452967 2753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7a14f10d0473380ee92d939da41cb" candidate_uuid: "94f4a49f47484c4d8fa4e588861ed76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7675e981c580478e9a85553fce6b1687"
I20260619 16:54:12.453056 2492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7a14f10d0473380ee92d939da41cb" candidate_uuid: "94f4a49f47484c4d8fa4e588861ed76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73fba737befc4e44838982e2097777c7"
I20260619 16:54:12.453172 2753 raft_consensus.cc:3037] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:12.453317 2492 raft_consensus.cc:3037] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:12.455564 2753 raft_consensus.cc:2454] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f4a49f47484c4d8fa4e588861ed76c in term 1.
I20260619 16:54:12.455577 2492 raft_consensus.cc:2454] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f4a49f47484c4d8fa4e588861ed76c in term 1.
I20260619 16:54:12.455978 2556 leader_election.cc:302] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7675e981c580478e9a85553fce6b1687, 94f4a49f47484c4d8fa4e588861ed76c; no voters:
I20260619 16:54:12.456290 2824 raft_consensus.cc:2781] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:12.456730 2824 raft_consensus.cc:686] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 1 LEADER]: Becoming Leader. State: Replica: 94f4a49f47484c4d8fa4e588861ed76c, State: Running, Role: LEADER
I20260619 16:54:12.457216 2824 consensus_queue.cc:237] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } }
I20260619 16:54:12.462256 2351 catalog_manager.cc:5495] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f4a49f47484c4d8fa4e588861ed76c (127.14.153.2). New cstate: current_term: 1 leader_uuid: "94f4a49f47484c4d8fa4e588861ed76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94f4a49f47484c4d8fa4e588861ed76c" member_type: VOTER last_known_addr { host: "127.14.153.2" port: 38379 } health_report { overall_health: HEALTHY } } }
I20260619 16:54:12.481784 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.476363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.14.153.2:38379)
0619 16:54:12.476547 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.476551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.476568 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:54:12.476726 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.476729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.476816 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.477019 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:12.477384 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.478191 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.479678 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.480084 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.480198 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.480593 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:12.481115 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:12.481235 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.481381 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.481465 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:54:12.490893 2753 raft_consensus.cc:1261] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [term 1 FOLLOWER]: Refusing update from remote peer 94f4a49f47484c4d8fa4e588861ed76c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:12.490895 2492 raft_consensus.cc:1261] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [term 1 FOLLOWER]: Refusing update from remote peer 94f4a49f47484c4d8fa4e588861ed76c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:12.492163 2827 consensus_queue.cc:1035] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73fba737befc4e44838982e2097777c7" member_type: VOTER last_known_addr { host: "127.14.153.1" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:12.492604 2824 consensus_queue.cc:1035] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7675e981c580478e9a85553fce6b1687" member_type: VOTER last_known_addr { host: "127.14.153.3" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:12.504139 2830 mvcc.cc:205] Tried to move back new op lower bound from 7298613462991564800 to 7298613462868828160. Current Snapshot: MvccSnapshot[applied={T|T < 7298613462991564800}]
I20260619 16:54:12.521968 2350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:54:12.522323 2350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:54:12.526662 2350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94d78e9c2735456595d0b857291f0077: Sending DeleteTablet for 3 replicas of tablet 30c7a14f10d0473380ee92d939da41cb
I20260619 16:54:12.527951 2733 tablet_service.cc:1514] Processing DeleteTablet for tablet 30c7a14f10d0473380ee92d939da41cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:12 UTC) from {username='slave'} at 127.0.0.1:58334
I20260619 16:54:12.528038 2603 tablet_service.cc:1514] Processing DeleteTablet for tablet 30c7a14f10d0473380ee92d939da41cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:12 UTC) from {username='slave'} at 127.0.0.1:42232
I20260619 16:54:12.528044 2472 tablet_service.cc:1514] Processing DeleteTablet for tablet 30c7a14f10d0473380ee92d939da41cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:54:12 UTC) from {username='slave'} at 127.0.0.1:54004
I20260619 16:54:12.534147 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 2410
I20260619 16:54:12.539556 2840 tablet_replica.cc:330] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: stopping tablet replica
I20260619 16:54:12.540668 2840 raft_consensus.cc:2229] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:54:12.541853 2840 raft_consensus.cc:2258] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:12.542887 2841 tablet_replica.cc:330] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: stopping tablet replica
I20260619 16:54:12.543629 2842 tablet_replica.cc:330] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: stopping tablet replica
I20260619 16:54:12.544128 2841 raft_consensus.cc:2229] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:54:12.544941 2841 raft_consensus.cc:2258] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:12.545557 2842 raft_consensus.cc:2229] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:54:12.546365 2840 ts_tablet_manager.cc:1905] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:12.546432 2842 raft_consensus.cc:2258] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:54:12.548758 2842 ts_tablet_manager.cc:1905] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:12.549675 2841 ts_tablet_manager.cc:1905] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:54:12.553587 2840 ts_tablet_manager.cc:1918] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:12.553871 2840 log.cc:1196] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-2/wal/wals/30c7a14f10d0473380ee92d939da41cb
I20260619 16:54:12.554678 2840 ts_tablet_manager.cc:1939] T 30c7a14f10d0473380ee92d939da41cb P 7675e981c580478e9a85553fce6b1687: Deleting consensus metadata
I20260619 16:54:12.556829 2841 ts_tablet_manager.cc:1918] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:12.556803 2335 catalog_manager.cc:4841] TS 7675e981c580478e9a85553fce6b1687 (127.14.153.3:35049): tablet 30c7a14f10d0473380ee92d939da41cb (table test-table [id=e93369c6e8694d16afc80b93597016c9]) successfully deleted
I20260619 16:54:12.557117 2841 log.cc:1196] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-1/wal/wals/30c7a14f10d0473380ee92d939da41cb
I20260619 16:54:12.557382 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.535543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.14.153.1:46845)
0619 16:54:12.535735 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.535742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.537082 (+ 1340us) server_negotiation.cc:407] Connection header received
0619 16:54:12.538198 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.538208 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.538357 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.538449 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:12.539419 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.540751 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.542830 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.543615 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.543750 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.555255 (+ 11505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:12.555267 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:12.555284 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:12.556130 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:12.556194 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:12.556222 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:12.556321 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.556854 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.556992 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:54:12.557803 2841 ts_tablet_manager.cc:1939] T 30c7a14f10d0473380ee92d939da41cb P 94f4a49f47484c4d8fa4e588861ed76c: Deleting consensus metadata
I20260619 16:54:12.558887 2532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:12.559893 2338 catalog_manager.cc:4841] TS 94f4a49f47484c4d8fa4e588861ed76c (127.14.153.2:38379): tablet 30c7a14f10d0473380ee92d939da41cb (table test-table [id=e93369c6e8694d16afc80b93597016c9]) successfully deleted
I20260619 16:54:12.652333 2842 ts_tablet_manager.cc:1918] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:54:12.652621 2842 log.cc:1196] T 30c7a14f10d0473380ee92d939da41cb P 73fba737befc4e44838982e2097777c7: Deleting WAL directory at /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887734944185-14948-0/minicluster-data/ts-0/wal/wals/30c7a14f10d0473380ee92d939da41cb
I20260619 16:54:12.652941 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 2410
I20260619 16:54:12.670989 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 2541
W20260619 16:54:12.670924 2336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.153.1:46845 (error 108)
I20260619 16:54:12.673115 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.672548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:46845 (local address 127.0.0.1:54022)
0619 16:54:12.672786 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.672918 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 16:54:12.673384 2336 catalog_manager.cc:4584] TS 73fba737befc4e44838982e2097777c7 (127.14.153.1:46845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30c7a14f10d0473380ee92d939da41cb: Network error: Client connection negotiation failed: client connection to 127.14.153.1:46845: connect: Connection refused (error 111)
I20260619 16:54:12.679176 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.671884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.14.153.2:38379)
0619 16:54:12.672070 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.672077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.672385 (+ 308us) server_negotiation.cc:407] Connection header received
0619 16:54:12.672564 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.672571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.672730 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.672858 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:12.673334 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.674237 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.675691 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.676168 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.676321 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.677492 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:12.677501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:12.677516 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:12.678336 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:12.678389 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:12.678411 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:12.678531 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.678668 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.678778 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:54:12.680063 2663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:12.727114 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.726529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:46845 (local address 127.0.0.1:54036)
0619 16:54:12.726752 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.726861 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 16:54:12.728164 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.727785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:46845 (local address 127.0.0.1:54052)
0619 16:54:12.727940 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.728024 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 16:54:12.788903 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.788347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:46845 (local address 127.0.0.1:54066)
0619 16:54:12.788619 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.788712 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 16:54:12.790004 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.789623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:46845 (local address 127.0.0.1:54082)
0619 16:54:12.789809 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.789890 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 16:54:12.796347 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 2541
I20260619 16:54:12.816323 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 2671
I20260619 16:54:12.825498 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.817122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.14.153.3:35049)
0619 16:54:12.817313 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:12.817318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:12.817337 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:54:12.817398 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:12.817402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:12.817480 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:12.817568 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:12.817956 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.818707 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.820002 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:12.820393 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:12.820485 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:12.824273 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:12.824283 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:12.824298 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:12.824902 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:12.824952 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:12.824972 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:12.825061 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:12.825171 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 16:54:12.825263 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:54:12.826267 2793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:12.870999 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.870276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:46845 (local address 127.0.0.1:54094)
0619 16:54:12.870669 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.870781 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260619 16:54:12.872246 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:12.871749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.1:46845 (local address 127.0.0.1:54110)
0619 16:54:12.872005 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:12.872112 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.1:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 16:54:12.916810 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 2671
I20260619 16:54:12.933895 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 2320
I20260619 16:54:12.934995 2381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:13.048578 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 2320
2026-06-19T16:54:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2778 ms)
[----------] 6 tests from EncryptionPolicyTest (21674 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:13 dist-test-slave-qb36 krb5kdc[2851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:13 dist-test-slave-qb36 krb5kdc[2851](info): set up 2 sockets
Jun 19 16:54:13 dist-test-slave-qb36 krb5kdc[2851](info): commencing operation
krb5kdc: starting...
W20260619 16:54:14.171705 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:14 dist-test-slave-qb36 krb5kdc[2851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:14.430037 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:14.766116 2864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:14.766412 2864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:14.766585 2864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:14.774022 2864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:14.774139 2864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:14.774231 2864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:14.774323 2864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:14.774430 2864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:14.784756 2864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:14.786217 2864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:14.787712 2864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:14.799762 2872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:14.799794 2869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:14.799773 2870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:14.800597 2864 server_base.cc:1034] running on GCE node
I20260619 16:54:14.801393 2864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:14.802834 2864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:14.804075 2864 hybrid_clock.cc:648] HybridClock initialized: now 1781888054804040 us; error 45 us; skew 500 ppm
I20260619 16:54:14.804487 2864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:14 dist-test-slave-qb36 krb5kdc[2851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:14.807941 2864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:14.809690 2864 webserver.cc:460] Webserver started at http://127.0.0.1:38623/ using document root <none> and password file <none>
I20260619 16:54:14.810454 2864 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:14.810627 2864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:14.810952 2864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:14.814090 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "4439c79d7b5f48ffb4e6f957f7692f0b"
format_stamp: "Formatted at 2026-06-19 16:54:14 on dist-test-slave-qb36"
server_key: "2eb131f08f96eea5262feb3d9f2f17b9"
server_key_iv: "ef33f9332da8f6d2abef5be2cc014afc"
server_key_version: "encryptionkey@0"
I20260619 16:54:14.814927 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "4439c79d7b5f48ffb4e6f957f7692f0b"
format_stamp: "Formatted at 2026-06-19 16:54:14 on dist-test-slave-qb36"
server_key: "2eb131f08f96eea5262feb3d9f2f17b9"
server_key_iv: "ef33f9332da8f6d2abef5be2cc014afc"
server_key_version: "encryptionkey@0"
I20260619 16:54:14.820394 2864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:54:14.824040 2879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:14.825403 2864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:14.825596 2864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "4439c79d7b5f48ffb4e6f957f7692f0b"
format_stamp: "Formatted at 2026-06-19 16:54:14 on dist-test-slave-qb36"
server_key: "2eb131f08f96eea5262feb3d9f2f17b9"
server_key_iv: "ef33f9332da8f6d2abef5be2cc014afc"
server_key_version: "encryptionkey@0"
I20260619 16:54:14.825811 2864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:14.845872 2864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:14.850399 2864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:14.850800 2864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:14.869649 2864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33529
I20260619 16:54:14.869701 2930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33529 every 8 connection(s)
I20260619 16:54:14.871107 2864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:14.874889 2931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:14.875200 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 2864
I20260619 16:54:14.875429 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:14.876222 14948 external_mini_cluster.cc:1428] Setting key 049b1bdaa5bcc48f0c05c117b5053d93
I20260619 16:54:14.889237 2931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b: Bootstrap starting.
Jun 19 16:54:14 dist-test-slave-qb36 krb5kdc[2851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:54:14.893133 2931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:14.894258 2931 log.cc:824] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:14.897241 2931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b: No bootstrap required, opened a new log
I20260619 16:54:14.900739 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:14.878182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44329 (local address 127.0.0.1:33529)
0619 16:54:14.879020 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:14.879076 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:14.879143 (+ 67us) server_negotiation.cc:407] Connection header received
0619 16:54:14.881439 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:14.881463 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:14.881860 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:14.882201 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:14.883992 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:14.885392 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:14.887265 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:14.888004 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:14.888168 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:14.893112 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:14.893124 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:14.893141 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:14.893186 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:14.896451 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:14.897307 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:14.897317 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:14.897328 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:14.897510 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:14.897991 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:14.897995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:14.897997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:14.898516 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:14.898879 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:14.899389 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 16:54:14.899721 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":410,"threads_started":1}
I20260619 16:54:14.904763 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:14.905510 2931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439c79d7b5f48ffb4e6f957f7692f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } }
I20260619 16:54:14.905891 2931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:14.906008 2931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439c79d7b5f48ffb4e6f957f7692f0b, State: Initialized, Role: FOLLOWER
I20260619 16:54:14.906628 2931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439c79d7b5f48ffb4e6f957f7692f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } }
I20260619 16:54:14.906862 2931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:14.906987 2931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:14.907238 2931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:14.909652 2931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439c79d7b5f48ffb4e6f957f7692f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } }
I20260619 16:54:14.910267 2931 leader_election.cc:302] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4439c79d7b5f48ffb4e6f957f7692f0b; no voters:
I20260619 16:54:14.910980 2931 leader_election.cc:288] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:14.911092 2941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:14.913085 2941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [term 1 LEADER]: Becoming Leader. State: Replica: 4439c79d7b5f48ffb4e6f957f7692f0b, State: Running, Role: LEADER
I20260619 16:54:14.913458 2931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:14.913758 2941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439c79d7b5f48ffb4e6f957f7692f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } }
I20260619 16:54:14.923448 2942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4439c79d7b5f48ffb4e6f957f7692f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439c79d7b5f48ffb4e6f957f7692f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } } }
I20260619 16:54:14.924161 2942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:14.924712 2943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4439c79d7b5f48ffb4e6f957f7692f0b. Latest consensus state: current_term: 1 leader_uuid: "4439c79d7b5f48ffb4e6f957f7692f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439c79d7b5f48ffb4e6f957f7692f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33529 } } }
I20260619 16:54:14.925137 2943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:14.931190 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:14.909777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59067 (local address 127.0.0.1:33529)
0619 16:54:14.909975 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:14.909984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:14.910477 (+ 493us) server_negotiation.cc:407] Connection header received
0619 16:54:14.910652 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:14.910656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:14.910756 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:14.910880 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:14.912869 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:14.914152 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:14.916479 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:14.917452 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:14.917613 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:14.921809 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:14.921817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:14.921821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:14.921868 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:14.925874 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:14.927310 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:14.927319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:14.927326 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:14.927511 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:14.928229 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:14.928237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:14.928243 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:14.928776 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:14.929177 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:14.930375 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0619 16:54:14.930619 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:54:14.936019 2950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:14.941656 2950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:14.950838 2950 catalog_manager.cc:1347] Generated new cluster ID: 16f46202efcc4062b054d37d174b4441
I20260619 16:54:14.950968 2950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:14.976272 2950 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:14.977644 2950 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:14.990473 2950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4439c79d7b5f48ffb4e6f957f7692f0b: Generated new TSK 0
I20260619 16:54:14.991549 2950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:15.007543 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 2864
I20260619 16:54:15.008836 2926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:15.104635 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 2864
W20260619 16:54:15.122254 2849 connection.cc:535] client connection to 127.0.0.1:33529 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33529): Connection reset by peer (error 104)
2026-06-19T16:54:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:15 dist-test-slave-qb36 krb5kdc[2963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:15 dist-test-slave-qb36 krb5kdc[2963](info): set up 2 sockets
Jun 19 16:54:15 dist-test-slave-qb36 krb5kdc[2963](info): commencing operation
krb5kdc: starting...
W20260619 16:54:16.219555 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:16 dist-test-slave-qb36 krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:16.478359 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:16.817415 2976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:16.817745 2976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:16.817925 2976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:16.825340 2976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:16.825459 2976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:16.825554 2976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:16.825671 2976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:16.825749 2976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:16.836391 2976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37185
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:16.837793 2976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:16.839156 2976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:16.851287 2981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:16.851260 2984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:16.851346 2982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:16.851936 2976 server_base.cc:1034] running on GCE node
I20260619 16:54:16.852878 2976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:16.854302 2976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:16.855538 2976 hybrid_clock.cc:648] HybridClock initialized: now 1781888056855467 us; error 55 us; skew 500 ppm
I20260619 16:54:16.855924 2976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:16 dist-test-slave-qb36 krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:16.859359 2976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:16.860947 2976 webserver.cc:460] Webserver started at http://127.0.0.1:36975/ using document root <none> and password file <none>
I20260619 16:54:16.861649 2976 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:16.861801 2976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:16.862104 2976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:16.865213 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "80dc4d4d2c2a448f9ccad430de5c056e"
format_stamp: "Formatted at 2026-06-19 16:54:16 on dist-test-slave-qb36"
server_key: "c180603a778a5c84642c0607e6485245"
server_key_iv: "a77e7cc5db4fe671f39a926271f0312c"
server_key_version: "encryptionkey@0"
I20260619 16:54:16.866009 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "80dc4d4d2c2a448f9ccad430de5c056e"
format_stamp: "Formatted at 2026-06-19 16:54:16 on dist-test-slave-qb36"
server_key: "c180603a778a5c84642c0607e6485245"
server_key_iv: "a77e7cc5db4fe671f39a926271f0312c"
server_key_version: "encryptionkey@0"
I20260619 16:54:16.871498 2976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 16:54:16.875296 2991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:16.876806 2976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:16.877000 2976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "80dc4d4d2c2a448f9ccad430de5c056e"
format_stamp: "Formatted at 2026-06-19 16:54:16 on dist-test-slave-qb36"
server_key: "c180603a778a5c84642c0607e6485245"
server_key_iv: "a77e7cc5db4fe671f39a926271f0312c"
server_key_version: "encryptionkey@0"
I20260619 16:54:16.877210 2976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:16.893841 2976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:16.898376 2976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:16.898669 2976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:16.917966 2976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37185
I20260619 16:54:16.917975 3042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37185 every 8 connection(s)
I20260619 16:54:16.919471 2976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:16.923346 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:16.924652 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 2976
I20260619 16:54:16.924855 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:16.925398 14948 external_mini_cluster.cc:1428] Setting key ebaa4a105da076ae4e062c2dcc62786f
I20260619 16:54:16.936411 3043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e: Bootstrap starting.
I20260619 16:54:16.938867 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:16.927402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45697 (local address 127.0.0.1:37185)
0619 16:54:16.928271 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:16.928286 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:16.928328 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:54:16.930058 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:16.930085 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:16.930511 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:16.930940 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:16.931502 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:16.933007 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:16.934342 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:16.935019 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:16.935271 (+ 252us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:16.936637 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:16.936645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:16.936660 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:16.937188 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:16.937263 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:16.937292 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:16.937460 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:16.937741 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:54:16.937924 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":385,"threads_started":1}
I20260619 16:54:16.939796 3043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:16.940925 3043 log.cc:824] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:16.942853 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:16.944334 3043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e: No bootstrap required, opened a new log
I20260619 16:54:16.955114 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:16.948157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52577 (local address 127.0.0.1:37185)
0619 16:54:16.948343 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:16.948353 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:16.948749 (+ 396us) server_negotiation.cc:407] Connection header received
0619 16:54:16.948898 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:16.948904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:16.948987 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:16.949094 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:16.949617 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:16.950803 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:16.952148 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:16.952880 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:16.953016 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:16.954129 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:16.954137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:16.954144 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:16.954171 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:16.954234 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:16.954257 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:16.954376 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:16.954681 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 16:54:16.954743 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:54:16.954957 3043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dc4d4d2c2a448f9ccad430de5c056e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37185 } }
I20260619 16:54:16.955580 3043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:16.955725 3043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80dc4d4d2c2a448f9ccad430de5c056e, State: Initialized, Role: FOLLOWER
I20260619 16:54:16.956430 3043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dc4d4d2c2a448f9ccad430de5c056e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37185 } }
I20260619 16:54:16.956701 3043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:16.956843 3043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:16.957034 3043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:16.959338 3043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dc4d4d2c2a448f9ccad430de5c056e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37185 } }
I20260619 16:54:16.959842 3043 leader_election.cc:302] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80dc4d4d2c2a448f9ccad430de5c056e; no voters:
I20260619 16:54:16.960479 3043 leader_election.cc:288] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:16.960613 3053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:16.961855 3053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [term 1 LEADER]: Becoming Leader. State: Replica: 80dc4d4d2c2a448f9ccad430de5c056e, State: Running, Role: LEADER
I20260619 16:54:16.962559 3053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dc4d4d2c2a448f9ccad430de5c056e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37185 } }
I20260619 16:54:16.962946 3043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:16.965510 3054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80dc4d4d2c2a448f9ccad430de5c056e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dc4d4d2c2a448f9ccad430de5c056e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37185 } } }
I20260619 16:54:16.965718 3055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80dc4d4d2c2a448f9ccad430de5c056e. Latest consensus state: current_term: 1 leader_uuid: "80dc4d4d2c2a448f9ccad430de5c056e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80dc4d4d2c2a448f9ccad430de5c056e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37185 } } }
I20260619 16:54:16.965975 3054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:16.966028 3055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:16.967958 3061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:16.973403 3061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:16.982882 3061 catalog_manager.cc:1347] Generated new cluster ID: fa96bc8a044d46aa8ce0d63f715584ed
I20260619 16:54:16.983069 3061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:17.000155 3061 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:17.001492 3061 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:17.022837 3061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80dc4d4d2c2a448f9ccad430de5c056e: Generated new TSK 0
I20260619 16:54:17.023864 3061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:17.031983 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 2976
W20260619 16:54:17.033109 3038 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45697
W20260619 16:54:17.033829 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 2976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:17.034024 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 2976
W20260619 16:54:17.050920 2961 connection.cc:535] client connection to 127.0.0.1:37185 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37185): Connection reset by peer (error 104)
2026-06-19T16:54:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:17 dist-test-slave-qb36 krb5kdc[3074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:17 dist-test-slave-qb36 krb5kdc[3074](info): set up 2 sockets
Jun 19 16:54:17 dist-test-slave-qb36 krb5kdc[3074](info): commencing operation
krb5kdc: starting...
W20260619 16:54:18.151576 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:18 dist-test-slave-qb36 krb5kdc[3074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:18.410354 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:18.746054 3087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:18.746374 3087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:18.746469 3087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:18.753916 3087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:18.754053 3087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:18.754103 3087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:18.754164 3087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:18.754202 3087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:18.764480 3087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:18.765793 3087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:18.767125 3087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:18.779940 3095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:18.779974 3093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:18.779882 3092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:18.781025 3087 server_base.cc:1034] running on GCE node
I20260619 16:54:18.781920 3087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:18.783556 3087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:18.784890 3087 hybrid_clock.cc:648] HybridClock initialized: now 1781888058784825 us; error 51 us; skew 500 ppm
I20260619 16:54:18.785343 3087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:18 dist-test-slave-qb36 krb5kdc[3074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:18.794675 3087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:18.796506 3087 webserver.cc:460] Webserver started at http://127.0.0.1:41237/ using document root <none> and password file <none>
I20260619 16:54:18.797284 3087 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:18.797448 3087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:18.797796 3087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:18.801366 3087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "23a4772d8d3a48e692dc9b283e34013a"
format_stamp: "Formatted at 2026-06-19 16:54:18 on dist-test-slave-qb36"
server_key: "65a44025c8135becd0a4c6a6a39b713e"
server_key_iv: "8b971b8fcd4e75700dbeb92e7d05acde"
server_key_version: "encryptionkey@0"
I20260619 16:54:18.802253 3087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "23a4772d8d3a48e692dc9b283e34013a"
format_stamp: "Formatted at 2026-06-19 16:54:18 on dist-test-slave-qb36"
server_key: "65a44025c8135becd0a4c6a6a39b713e"
server_key_iv: "8b971b8fcd4e75700dbeb92e7d05acde"
server_key_version: "encryptionkey@0"
I20260619 16:54:18.808264 3087 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 16:54:18.812341 3102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:18.813930 3087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:54:18.814154 3087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "23a4772d8d3a48e692dc9b283e34013a"
format_stamp: "Formatted at 2026-06-19 16:54:18 on dist-test-slave-qb36"
server_key: "65a44025c8135becd0a4c6a6a39b713e"
server_key_iv: "8b971b8fcd4e75700dbeb92e7d05acde"
server_key_version: "encryptionkey@0"
I20260619 16:54:18.814392 3087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:18.820815 3087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:18.825412 3087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:18.825752 3087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:18.845490 3087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42551
I20260619 16:54:18.845510 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42551 every 8 connection(s)
I20260619 16:54:18.846990 3087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:18.850765 3154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:18.855844 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 3087
I20260619 16:54:18.856032 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:18.856520 14948 external_mini_cluster.cc:1428] Setting key 4f8e6a0fe23971c6fa8eec8c89b15b14
I20260619 16:54:18.862176 3154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a: Bootstrap starting.
I20260619 16:54:18.864632 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:18.858451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57595 (local address 127.0.0.1:42551)
0619 16:54:18.859363 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:18.859376 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:18.859422 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:54:18.861159 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:18.861184 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:18.861547 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:18.861892 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:18.862969 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:18.862975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:18.862989 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:18.863007 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:18.863127 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:18.863154 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:18.863337 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:18.863439 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 16:54:18.863698 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":414,"threads_started":1}
I20260619 16:54:18.865986 3154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:18.867129 3154 log.cc:824] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:18.868525 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:18.870072 3154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a: No bootstrap required, opened a new log
I20260619 16:54:18.875989 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:18.873376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36149 (local address 127.0.0.1:42551)
0619 16:54:18.873594 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:18.873601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:18.873964 (+ 363us) server_negotiation.cc:407] Connection header received
0619 16:54:18.874112 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:18.874120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:18.874193 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:18.874322 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:18.875315 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:18.875321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:18.875326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:18.875345 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:18.875401 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:18.875421 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:18.875565 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:18.875628 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 16:54:18.875683 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:54:18.878679 3154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a4772d8d3a48e692dc9b283e34013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260619 16:54:18.879139 3154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:18.879271 3154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a4772d8d3a48e692dc9b283e34013a, State: Initialized, Role: FOLLOWER
I20260619 16:54:18.879839 3154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a4772d8d3a48e692dc9b283e34013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260619 16:54:18.880067 3154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:18.880178 3154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:18.880312 3154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:18.882498 3154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a4772d8d3a48e692dc9b283e34013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260619 16:54:18.883016 3154 leader_election.cc:302] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23a4772d8d3a48e692dc9b283e34013a; no voters:
I20260619 16:54:18.883679 3154 leader_election.cc:288] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:18.883828 3164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:18.884518 3164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [term 1 LEADER]: Becoming Leader. State: Replica: 23a4772d8d3a48e692dc9b283e34013a, State: Running, Role: LEADER
I20260619 16:54:18.885156 3164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a4772d8d3a48e692dc9b283e34013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260619 16:54:18.885968 3154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:18.888077 3166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23a4772d8d3a48e692dc9b283e34013a. Latest consensus state: current_term: 1 leader_uuid: "23a4772d8d3a48e692dc9b283e34013a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a4772d8d3a48e692dc9b283e34013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } } }
I20260619 16:54:18.888087 3165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23a4772d8d3a48e692dc9b283e34013a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23a4772d8d3a48e692dc9b283e34013a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } } }
I20260619 16:54:18.888545 3165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:18.888545 3166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:18.890419 3171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:18.897619 3171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:18.907244 3171 catalog_manager.cc:1347] Generated new cluster ID: 1ed20a57316a4bfb8881e9a21af3dbd7
I20260619 16:54:18.907377 3171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:18.926261 3171 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:18.926752 3171 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:18.938314 3171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23a4772d8d3a48e692dc9b283e34013a: Generated new TSK 0
I20260619 16:54:18.939229 3171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:18.950474 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3087
W20260619 16:54:18.951686 3149 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57595
W20260619 16:54:18.952454 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3087: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:18.952605 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 3087
2026-06-19T16:54:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:19 dist-test-slave-qb36 krb5kdc[3185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:19 dist-test-slave-qb36 krb5kdc[3185](info): set up 2 sockets
Jun 19 16:54:19 dist-test-slave-qb36 krb5kdc[3185](info): commencing operation
krb5kdc: starting...
W20260619 16:54:20.071350 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:20 dist-test-slave-qb36 krb5kdc[3185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:20.333926 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:20.673710 3198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:20.674158 3198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:20.674331 3198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:20.682318 3198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:20.682454 3198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:20.682503 3198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:20.682633 3198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:20.682685 3198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:20.693392 3198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:20.694798 3198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:20.696293 3198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:20.709017 3203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:20.709049 3204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:20.709084 3206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:20.710145 3198 server_base.cc:1034] running on GCE node
I20260619 16:54:20.711001 3198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:20.712563 3198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:20.713820 3198 hybrid_clock.cc:648] HybridClock initialized: now 1781888060713771 us; error 44 us; skew 500 ppm
I20260619 16:54:20.714205 3198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:20 dist-test-slave-qb36 krb5kdc[3185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:20.717455 3198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:20.719122 3198 webserver.cc:460] Webserver started at http://127.0.0.1:43929/ using document root <none> and password file <none>
I20260619 16:54:20.719837 3198 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:20.719983 3198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:20.720314 3198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:20.723529 3198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "887bab28fdab4895ad4ddd621507d42f"
format_stamp: "Formatted at 2026-06-19 16:54:20 on dist-test-slave-qb36"
server_key: "e6561151a2e13ff5f9d47b8acae2e94f"
server_key_iv: "64bdad7d85c781b67908fb29f1dcefe3"
server_key_version: "encryptionkey@0"
I20260619 16:54:20.724375 3198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "887bab28fdab4895ad4ddd621507d42f"
format_stamp: "Formatted at 2026-06-19 16:54:20 on dist-test-slave-qb36"
server_key: "e6561151a2e13ff5f9d47b8acae2e94f"
server_key_iv: "64bdad7d85c781b67908fb29f1dcefe3"
server_key_version: "encryptionkey@0"
I20260619 16:54:20.729871 3198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 16:54:20.733469 3213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:20.734858 3198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:54:20.735118 3198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "887bab28fdab4895ad4ddd621507d42f"
format_stamp: "Formatted at 2026-06-19 16:54:20 on dist-test-slave-qb36"
server_key: "e6561151a2e13ff5f9d47b8acae2e94f"
server_key_iv: "64bdad7d85c781b67908fb29f1dcefe3"
server_key_version: "encryptionkey@0"
I20260619 16:54:20.735327 3198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:20.757907 3198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:20.762310 3198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:20.762609 3198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:20.781308 3198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44759
I20260619 16:54:20.781343 3264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44759 every 8 connection(s)
I20260619 16:54:20.782757 3198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:20.786732 3265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:20.790165 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 3198
I20260619 16:54:20.790364 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:20.790828 14948 external_mini_cluster.cc:1428] Setting key cc7c3b7b88cb15dfd3fe51a0e0c8c365
I20260619 16:54:20.799866 3265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f: Bootstrap starting.
I20260619 16:54:20.803864 3265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:20.805255 3265 log.cc:824] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:54:20 dist-test-slave-qb36 krb5kdc[3185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:54:20.808560 3265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f: No bootstrap required, opened a new log
I20260619 16:54:20.815418 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:20.792904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54347 (local address 127.0.0.1:44759)
0619 16:54:20.793729 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:20.793741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:20.793789 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:54:20.795591 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:20.795622 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:20.796062 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:20.796414 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:20.798241 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:20.799714 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:20.801463 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:20.802188 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:20.802323 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:20.807500 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:20.807508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:20.807524 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:20.807557 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:20.810808 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:20.811747 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:20.811754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:20.811764 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:20.811939 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:20.812361 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:20.812364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:20.812367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:20.812929 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:20.813158 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:20.813720 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 16:54:20.814232 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":390,"threads_started":1}
I20260619 16:54:20.817914 3265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887bab28fdab4895ad4ddd621507d42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260619 16:54:20.818423 3265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:20.818536 3265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887bab28fdab4895ad4ddd621507d42f, State: Initialized, Role: FOLLOWER
I20260619 16:54:20.819290 3265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887bab28fdab4895ad4ddd621507d42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260619 16:54:20.819591 3265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:20.819734 3265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:20.819810 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:20.819912 3265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:20.822108 3265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887bab28fdab4895ad4ddd621507d42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260619 16:54:20.822641 3265 leader_election.cc:302] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 887bab28fdab4895ad4ddd621507d42f; no voters:
I20260619 16:54:20.823453 3265 leader_election.cc:288] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:20.823551 3273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:20.825510 3273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [term 1 LEADER]: Becoming Leader. State: Replica: 887bab28fdab4895ad4ddd621507d42f, State: Running, Role: LEADER
I20260619 16:54:20.826079 3265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:20.826177 3273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887bab28fdab4895ad4ddd621507d42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } }
I20260619 16:54:20.829520 3275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "887bab28fdab4895ad4ddd621507d42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887bab28fdab4895ad4ddd621507d42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } } }
I20260619 16:54:20.829952 3275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:20.830406 3276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 887bab28fdab4895ad4ddd621507d42f. Latest consensus state: current_term: 1 leader_uuid: "887bab28fdab4895ad4ddd621507d42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887bab28fdab4895ad4ddd621507d42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44759 } } }
I20260619 16:54:20.830721 3276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:20.833379 3283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:20.838725 3283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:20.846517 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:20.825036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33357 (local address 127.0.0.1:44759)
0619 16:54:20.825188 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:20.825194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:20.825574 (+ 380us) server_negotiation.cc:407] Connection header received
0619 16:54:20.825842 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:20.825849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:20.825918 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:20.826025 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:20.827733 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:20.829039 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:20.830969 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:20.831907 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:20.832053 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:20.836111 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:20.836121 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:20.836129 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:20.836186 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:20.841008 (+ 4822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:20.842206 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:20.842216 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:20.842223 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:20.842417 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:20.843016 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:20.843057 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:20.843063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:20.843664 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:20.844125 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:20.845630 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0619 16:54:20.845969 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:54:20.852077 3283 catalog_manager.cc:1347] Generated new cluster ID: 7f33ba7dac1f47f7998b1638e1a8f2e4
I20260619 16:54:20.852216 3283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:20.862726 3283 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:20.864203 3283 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:20.872931 3283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 887bab28fdab4895ad4ddd621507d42f: Generated new TSK 0
I20260619 16:54:20.873858 3283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:20.879868 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3198
I20260619 16:54:20.881016 3260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:20.972679 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 3198
2026-06-19T16:54:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:21 dist-test-slave-qb36 krb5kdc[3297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:21 dist-test-slave-qb36 krb5kdc[3297](info): set up 2 sockets
Jun 19 16:54:21 dist-test-slave-qb36 krb5kdc[3297](info): commencing operation
krb5kdc: starting...
W20260619 16:54:22.087584 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:22 dist-test-slave-qb36 krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:22.346360 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:22.676532 3310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:22.676856 3310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:22.676951 3310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:22.685786 3310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:22.685911 3310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:22.685958 3310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:22.686025 3310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:22.686069 3310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:22.696491 3310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:22.697762 3310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:22.699014 3310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:22.711047 3318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:22.711162 3316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:22.711009 3315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:22.716660 3310 server_base.cc:1034] running on GCE node
I20260619 16:54:22.717555 3310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:22.719020 3310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:22.720304 3310 hybrid_clock.cc:648] HybridClock initialized: now 1781888062720234 us; error 66 us; skew 500 ppm
I20260619 16:54:22.720676 3310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:22 dist-test-slave-qb36 krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:22.723945 3310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:22.725498 3310 webserver.cc:460] Webserver started at http://127.0.0.1:34201/ using document root <none> and password file <none>
I20260619 16:54:22.726207 3310 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:22.726310 3310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:22.726559 3310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:22.729621 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "c670a3a97c5b4660ad1e6eb5fa585125"
format_stamp: "Formatted at 2026-06-19 16:54:22 on dist-test-slave-qb36"
server_key: "13b76a5adf5f5d9aed4dc839ddf40f81"
server_key_iv: "cb929b37dff6da35ea428c2ef114e41b"
server_key_version: "encryptionkey@0"
I20260619 16:54:22.730357 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "c670a3a97c5b4660ad1e6eb5fa585125"
format_stamp: "Formatted at 2026-06-19 16:54:22 on dist-test-slave-qb36"
server_key: "13b76a5adf5f5d9aed4dc839ddf40f81"
server_key_iv: "cb929b37dff6da35ea428c2ef114e41b"
server_key_version: "encryptionkey@0"
I20260619 16:54:22.735471 3310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 16:54:22.738900 3325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:22.740289 3310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:22.740471 3310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "c670a3a97c5b4660ad1e6eb5fa585125"
format_stamp: "Formatted at 2026-06-19 16:54:22 on dist-test-slave-qb36"
server_key: "13b76a5adf5f5d9aed4dc839ddf40f81"
server_key_iv: "cb929b37dff6da35ea428c2ef114e41b"
server_key_version: "encryptionkey@0"
I20260619 16:54:22.740674 3310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:22.765890 3310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:22.769937 3310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:22.770236 3310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:22.789017 3310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42773
I20260619 16:54:22.789034 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42773 every 8 connection(s)
I20260619 16:54:22.790470 3310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:22.791843 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 3310
I20260619 16:54:22.792016 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:22.792574 14948 external_mini_cluster.cc:1428] Setting key 399d4070f57577b0c767e213f7de25ab
I20260619 16:54:22.795439 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:22.805496 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:22.794484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42607 (local address 127.0.0.1:42773)
0619 16:54:22.795343 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:22.795355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:22.795394 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:54:22.797107 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:22.797129 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:22.797521 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:22.797925 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:22.798474 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:22.799834 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:22.801146 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:22.801803 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:22.801993 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:22.803252 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:22.803260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:22.803273 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:22.803834 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:22.803911 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:22.803934 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:22.804077 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:22.804311 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 16:54:22.804559 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":336,"threads_started":1}
I20260619 16:54:22.806716 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125: Bootstrap starting.
I20260619 16:54:22.809510 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:22.810334 3377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:22.811415 3377 log.cc:824] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:22.814810 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125: No bootstrap required, opened a new log
I20260619 16:54:22.820292 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:22.814119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33295 (local address 127.0.0.1:42773)
0619 16:54:22.814274 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:22.814279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:22.814784 (+ 505us) server_negotiation.cc:407] Connection header received
0619 16:54:22.815004 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:22.815008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:22.815129 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:22.815230 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:22.815714 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:22.816465 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:22.817762 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:22.818243 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:22.818403 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:22.819528 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:22.819533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:22.819536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:22.819555 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:22.819592 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:22.819610 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:22.819711 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:22.819870 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 16:54:22.819966 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:54:22.823292 3377 raft_consensus.cc:348] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c670a3a97c5b4660ad1e6eb5fa585125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } }
I20260619 16:54:22.823721 3377 raft_consensus.cc:374] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:22.823838 3377 raft_consensus.cc:729] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c670a3a97c5b4660ad1e6eb5fa585125, State: Initialized, Role: FOLLOWER
I20260619 16:54:22.824437 3377 consensus_queue.cc:260] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c670a3a97c5b4660ad1e6eb5fa585125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } }
I20260619 16:54:22.824688 3377 raft_consensus.cc:388] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:22.824839 3377 raft_consensus.cc:482] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:22.825021 3377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:22.827590 3377 raft_consensus.cc:504] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c670a3a97c5b4660ad1e6eb5fa585125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } }
I20260619 16:54:22.828220 3377 leader_election.cc:302] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c670a3a97c5b4660ad1e6eb5fa585125; no voters:
I20260619 16:54:22.828806 3377 leader_election.cc:288] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:22.828955 3387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:22.829682 3387 raft_consensus.cc:686] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [term 1 LEADER]: Becoming Leader. State: Replica: c670a3a97c5b4660ad1e6eb5fa585125, State: Running, Role: LEADER
I20260619 16:54:22.830256 3387 consensus_queue.cc:237] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c670a3a97c5b4660ad1e6eb5fa585125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } }
I20260619 16:54:22.831447 3377 sys_catalog.cc:564] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:22.833630 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c670a3a97c5b4660ad1e6eb5fa585125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c670a3a97c5b4660ad1e6eb5fa585125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } } }
I20260619 16:54:22.833525 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c670a3a97c5b4660ad1e6eb5fa585125. Latest consensus state: current_term: 1 leader_uuid: "c670a3a97c5b4660ad1e6eb5fa585125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c670a3a97c5b4660ad1e6eb5fa585125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } } }
I20260619 16:54:22.834055 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:22.834229 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:22.837355 3395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:22.842725 3395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:22.851763 3395 catalog_manager.cc:1347] Generated new cluster ID: 63884cdb1db24523b4a1e9b9c6ccba33
I20260619 16:54:22.851907 3395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:22.867511 3395 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:22.868937 3395 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:22.881417 3395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c670a3a97c5b4660ad1e6eb5fa585125: Generated new TSK 0
I20260619 16:54:22.882320 3395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:22.890973 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3310
W20260619 16:54:22.892135 3372 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42607
W20260619 16:54:22.892874 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3310: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:22.893030 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 3310
2026-06-19T16:54:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:22 dist-test-slave-qb36 krb5kdc[3408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:22 dist-test-slave-qb36 krb5kdc[3408](info): set up 2 sockets
Jun 19 16:54:22 dist-test-slave-qb36 krb5kdc[3408](info): commencing operation
krb5kdc: starting...
W20260619 16:54:24.019558 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:24 dist-test-slave-qb36 krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:24.276556 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:24.613626 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:24.613930 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:24.614109 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:24.621490 3421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:24.621626 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:24.621759 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:24.621896 3421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:24.621965 3421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:24.632052 3421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:24.633464 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:24.634827 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:24.647308 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:24.647310 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:24.647333 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:24.648180 3421 server_base.cc:1034] running on GCE node
I20260619 16:54:24.649037 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:24.650513 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:24.651748 3421 hybrid_clock.cc:648] HybridClock initialized: now 1781888064651711 us; error 38 us; skew 500 ppm
I20260619 16:54:24.652117 3421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:24 dist-test-slave-qb36 krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:24.658702 3421 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:24.660427 3421 webserver.cc:460] Webserver started at http://127.0.0.1:34845/ using document root <none> and password file <none>
I20260619 16:54:24.661154 3421 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:24.661304 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:24.661623 3421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:24.664752 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "701c7b60236647ea8f576327a71b051f"
format_stamp: "Formatted at 2026-06-19 16:54:24 on dist-test-slave-qb36"
server_key: "aca197a5de00ab8f20b9a23046a3754b"
server_key_iv: "df559f19818c07645eafa2264c839cd5"
server_key_version: "encryptionkey@0"
I20260619 16:54:24.665575 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "701c7b60236647ea8f576327a71b051f"
format_stamp: "Formatted at 2026-06-19 16:54:24 on dist-test-slave-qb36"
server_key: "aca197a5de00ab8f20b9a23046a3754b"
server_key_iv: "df559f19818c07645eafa2264c839cd5"
server_key_version: "encryptionkey@0"
I20260619 16:54:24.670820 3421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:54:24.674387 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:24.675868 3421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:24.676064 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "701c7b60236647ea8f576327a71b051f"
format_stamp: "Formatted at 2026-06-19 16:54:24 on dist-test-slave-qb36"
server_key: "aca197a5de00ab8f20b9a23046a3754b"
server_key_iv: "df559f19818c07645eafa2264c839cd5"
server_key_version: "encryptionkey@0"
I20260619 16:54:24.676281 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:24.682416 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:24.686942 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:24.687289 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:24.707075 3421 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35723
I20260619 16:54:24.707127 3487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35723 every 8 connection(s)
I20260619 16:54:24.708531 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:24.711973 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 3421
I20260619 16:54:24.712152 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:24.712679 14948 external_mini_cluster.cc:1428] Setting key 868bbd8ff42a81a50a93881a6c895f61
I20260619 16:54:24.712934 3488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:24.721292 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:24.714546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49843 (local address 127.0.0.1:35723)
0619 16:54:24.715408 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:24.715421 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:24.715465 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:54:24.717712 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:24.717737 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:24.718109 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:24.718482 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:24.719633 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:24.719639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:24.719683 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:24.719698 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:24.719757 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:24.719787 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:24.719973 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:24.720070 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 16:54:24.720306 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":420,"threads_started":1}
I20260619 16:54:24.725131 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:24.726770 3488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f: Bootstrap starting.
I20260619 16:54:24.730863 3488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:24.731977 3488 log.cc:824] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:24.733422 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:24.730374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59035 (local address 127.0.0.1:35723)
0619 16:54:24.730571 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:24.730577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:24.731061 (+ 484us) server_negotiation.cc:407] Connection header received
0619 16:54:24.731329 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:24.731336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:24.731388 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:24.731496 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:24.732571 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:24.732578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:24.732583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:24.732603 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:24.732655 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:24.732681 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:24.732820 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:24.732956 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:54:24.733024 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:54:24.735203 3488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f: No bootstrap required, opened a new log
I20260619 16:54:24.742347 3488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c7b60236647ea8f576327a71b051f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } }
I20260619 16:54:24.742735 3488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:24.742843 3488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701c7b60236647ea8f576327a71b051f, State: Initialized, Role: FOLLOWER
I20260619 16:54:24.743456 3488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c7b60236647ea8f576327a71b051f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } }
I20260619 16:54:24.743690 3488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:24.743832 3488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:24.744014 3488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:24.746145 3488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c7b60236647ea8f576327a71b051f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } }
I20260619 16:54:24.746636 3488 leader_election.cc:302] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 701c7b60236647ea8f576327a71b051f; no voters:
I20260619 16:54:24.747303 3488 leader_election.cc:288] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:24.747423 3498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:24.748153 3498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [term 1 LEADER]: Becoming Leader. State: Replica: 701c7b60236647ea8f576327a71b051f, State: Running, Role: LEADER
I20260619 16:54:24.748683 3498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c7b60236647ea8f576327a71b051f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } }
I20260619 16:54:24.749346 3488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:24.751648 3500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 701c7b60236647ea8f576327a71b051f. Latest consensus state: current_term: 1 leader_uuid: "701c7b60236647ea8f576327a71b051f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c7b60236647ea8f576327a71b051f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } } }
I20260619 16:54:24.752127 3500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:24.753906 3499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "701c7b60236647ea8f576327a71b051f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701c7b60236647ea8f576327a71b051f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } } }
I20260619 16:54:24.754195 3499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:24.754361 3507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:24.759624 3507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:24.769531 3507 catalog_manager.cc:1347] Generated new cluster ID: 45d9dac02e6f4aed8813ded7e5ec2af9
I20260619 16:54:24.769677 3507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:24.788784 3507 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:24.789260 3507 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:24.794941 3507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 701c7b60236647ea8f576327a71b051f: Generated new TSK 0
I20260619 16:54:24.796103 3507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:24.804749 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3421
W20260619 16:54:24.806041 3483 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49843
W20260619 16:54:24.806855 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3421: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:24.807041 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 3421
2026-06-19T16:54:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1916 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:24 dist-test-slave-qb36 krb5kdc[3519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:24 dist-test-slave-qb36 krb5kdc[3519](info): set up 2 sockets
Jun 19 16:54:24 dist-test-slave-qb36 krb5kdc[3519](info): commencing operation
krb5kdc: starting...
W20260619 16:54:25.911540 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:26 dist-test-slave-qb36 krb5kdc[3519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:26.168464 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:26.502965 3532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:26.503386 3532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:26.503551 3532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:26.510795 3532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:26.510962 3532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:26.511096 3532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:26.511193 3532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:26.511283 3532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:26.521121 3532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:26.522491 3532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:26.523860 3532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:26.536626 3537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:26.536659 3540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:26.536585 3538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:26.537581 3532 server_base.cc:1034] running on GCE node
I20260619 16:54:26.538370 3532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:26.539903 3532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:26.541103 3532 hybrid_clock.cc:648] HybridClock initialized: now 1781888066541062 us; error 33 us; skew 500 ppm
I20260619 16:54:26.541467 3532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:26 dist-test-slave-qb36 krb5kdc[3519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:26.544673 3532 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:26.546109 3532 webserver.cc:460] Webserver started at http://127.0.0.1:43167/ using document root <none> and password file <none>
I20260619 16:54:26.546797 3532 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:26.546942 3532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:26.547278 3532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:26.550203 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "ba63daffceb34bedb2eac79e02864a64"
format_stamp: "Formatted at 2026-06-19 16:54:26 on dist-test-slave-qb36"
server_key: "dcadee649735c5ce4092fa3fdd6007af"
server_key_iv: "3842243448afe6e7131a387cd8a6f585"
server_key_version: "encryptionkey@0"
I20260619 16:54:26.550978 3532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "ba63daffceb34bedb2eac79e02864a64"
format_stamp: "Formatted at 2026-06-19 16:54:26 on dist-test-slave-qb36"
server_key: "dcadee649735c5ce4092fa3fdd6007af"
server_key_iv: "3842243448afe6e7131a387cd8a6f585"
server_key_version: "encryptionkey@0"
I20260619 16:54:26.556331 3532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:54:26.559865 3547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:26.561216 3532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:26.561410 3532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "ba63daffceb34bedb2eac79e02864a64"
format_stamp: "Formatted at 2026-06-19 16:54:26 on dist-test-slave-qb36"
server_key: "dcadee649735c5ce4092fa3fdd6007af"
server_key_iv: "3842243448afe6e7131a387cd8a6f585"
server_key_version: "encryptionkey@0"
I20260619 16:54:26.561614 3532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:26.574759 3532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:26.579329 3532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:26.579631 3532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:26.598405 3532 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32981
I20260619 16:54:26.598433 3598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32981 every 8 connection(s)
I20260619 16:54:26.599990 3532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:26.602969 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 3532
I20260619 16:54:26.603204 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:26.603645 14948 external_mini_cluster.cc:1428] Setting key f687c44ebd1fefe46ab8d015f74a2d85
I20260619 16:54:26.603924 3599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:26.618510 3599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64: Bootstrap starting.
Jun 19 16:54:26 dist-test-slave-qb36 krb5kdc[3519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:54:26.622694 3599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:26.624290 3599 log.cc:824] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:26.627974 3599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64: No bootstrap required, opened a new log
I20260619 16:54:26.629285 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:26.605348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:35283 (local address 127.0.0.1:32981)
0619 16:54:26.606083 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:26.606094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:26.606136 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:54:26.608434 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:26.608456 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:26.608870 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:26.609185 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:26.611009 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:26.612346 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:26.614100 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:26.614785 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:26.614913 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:26.619979 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:26.619995 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:26.620016 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:26.620085 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:26.624298 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:26.625337 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:26.625343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:26.625350 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:26.625524 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:26.626106 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:26.626114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:26.626119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:26.626816 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:26.627179 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:26.627556 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 16:54:26.627887 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":340,"threads_started":1}
I20260619 16:54:26.633230 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:26.636420 3599 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba63daffceb34bedb2eac79e02864a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260619 16:54:26.636801 3599 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:26.636901 3599 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba63daffceb34bedb2eac79e02864a64, State: Initialized, Role: FOLLOWER
I20260619 16:54:26.637573 3599 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba63daffceb34bedb2eac79e02864a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260619 16:54:26.637903 3599 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:26.638062 3599 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:26.638266 3599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:26.640966 3599 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba63daffceb34bedb2eac79e02864a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260619 16:54:26.641561 3599 leader_election.cc:302] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba63daffceb34bedb2eac79e02864a64; no voters:
I20260619 16:54:26.642290 3599 leader_election.cc:288] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:26.642431 3609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:26.645104 3599 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:26.646973 3609 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [term 1 LEADER]: Becoming Leader. State: Replica: ba63daffceb34bedb2eac79e02864a64, State: Running, Role: LEADER
I20260619 16:54:26.647867 3609 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba63daffceb34bedb2eac79e02864a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260619 16:54:26.651233 3611 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba63daffceb34bedb2eac79e02864a64. Latest consensus state: current_term: 1 leader_uuid: "ba63daffceb34bedb2eac79e02864a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba63daffceb34bedb2eac79e02864a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } } }
I20260619 16:54:26.651723 3611 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:26.652595 3610 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba63daffceb34bedb2eac79e02864a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba63daffceb34bedb2eac79e02864a64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } } }
I20260619 16:54:26.653460 3610 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:26.658289 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:26.637851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:41213 (local address 127.0.0.1:32981)
0619 16:54:26.638056 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:26.638063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:26.638661 (+ 598us) server_negotiation.cc:407] Connection header received
0619 16:54:26.638792 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:26.638797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:26.638884 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:26.638999 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:26.640459 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:26.641650 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:26.643340 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:26.643786 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:26.643924 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:26.646531 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:26.646537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:26.646540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:26.646573 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:26.650562 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:26.651648 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:26.651654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:26.651657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:26.651835 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:26.652455 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:26.652459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:26.652462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:26.655982 (+ 3520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:26.656337 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:26.657513 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0619 16:54:26.657780 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:54:26.664089 3617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:26.670583 3617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:26.680140 3617 catalog_manager.cc:1347] Generated new cluster ID: 88a94370b7e5419bae70f42f57d3b333
I20260619 16:54:26.680264 3617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:26.700538 3617 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:26.701864 3617 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:26.708424 3617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba63daffceb34bedb2eac79e02864a64: Generated new TSK 0
I20260619 16:54:26.709234 3617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:26.716642 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3532
I20260619 16:54:26.717965 3594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:26.808553 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 3532
W20260619 16:54:26.825994 3517 connection.cc:535] client connection to 127.0.0.1:32981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32981): Connection reset by peer (error 104)
2026-06-19T16:54:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:26 dist-test-slave-qb36 krb5kdc[3632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:26 dist-test-slave-qb36 krb5kdc[3632](info): set up 2 sockets
Jun 19 16:54:26 dist-test-slave-qb36 krb5kdc[3632](info): commencing operation
krb5kdc: starting...
W20260619 16:54:27.927603 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:28 dist-test-slave-qb36 krb5kdc[3632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:28.191112 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:28.526561 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:28.526933 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:28.527118 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:28.534471 3645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:28.534649 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:28.534745 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:28.534837 3645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:28.534905 3645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:28.545241 3645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:28.546590 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:28.547969 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:28.560082 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:28.560004 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:28.560045 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:28.561157 3645 server_base.cc:1034] running on GCE node
I20260619 16:54:28.561985 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:28.563525 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:28.564759 3645 hybrid_clock.cc:648] HybridClock initialized: now 1781888068564736 us; error 75 us; skew 500 ppm
I20260619 16:54:28.565119 3645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:28 dist-test-slave-qb36 krb5kdc[3632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:28.568269 3645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:28.569730 3645 webserver.cc:460] Webserver started at http://127.0.0.1:35839/ using document root <none> and password file <none>
I20260619 16:54:28.570385 3645 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:28.570485 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:28.570737 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:28.573777 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "83dafe6349cb4f9e95b5d2403a934a23"
format_stamp: "Formatted at 2026-06-19 16:54:28 on dist-test-slave-qb36"
server_key: "a892c35c8b19b63b3146a4902df1ebc8"
server_key_iv: "103b8c57735866f3b5ab228774112bc2"
server_key_version: "encryptionkey@0"
I20260619 16:54:28.574519 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "83dafe6349cb4f9e95b5d2403a934a23"
format_stamp: "Formatted at 2026-06-19 16:54:28 on dist-test-slave-qb36"
server_key: "a892c35c8b19b63b3146a4902df1ebc8"
server_key_iv: "103b8c57735866f3b5ab228774112bc2"
server_key_version: "encryptionkey@0"
I20260619 16:54:28.580116 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 16:54:28.583799 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:28.585446 3645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:28.585654 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "83dafe6349cb4f9e95b5d2403a934a23"
format_stamp: "Formatted at 2026-06-19 16:54:28 on dist-test-slave-qb36"
server_key: "a892c35c8b19b63b3146a4902df1ebc8"
server_key_iv: "103b8c57735866f3b5ab228774112bc2"
server_key_version: "encryptionkey@0"
I20260619 16:54:28.585873 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:28.612681 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:28.617067 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:28.617396 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:28.636782 3645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44497
I20260619 16:54:28.636873 3711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44497 every 8 connection(s)
I20260619 16:54:28.638199 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:28.642071 3712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:28.647217 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 3645
I20260619 16:54:28.647392 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:28.647894 14948 external_mini_cluster.cc:1428] Setting key 82b8e976a1339c111b6c8eba07dbc1e2
I20260619 16:54:28.653431 3712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23: Bootstrap starting.
I20260619 16:54:28.657115 3712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:28.658380 3712 log.cc:824] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:28.661511 3712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23: No bootstrap required, opened a new log
I20260619 16:54:28.661651 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:28.649728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:46469 (local address 127.0.0.1:44497)
0619 16:54:28.650539 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:28.650550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:28.650596 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:54:28.652368 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:28.652391 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:28.652808 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:28.653130 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:28.653733 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:28.655129 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:28.656928 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:28.657663 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:28.657794 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:28.659211 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:28.659221 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:28.659241 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:28.659977 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:28.660047 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:28.660076 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:28.660192 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:28.660495 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 16:54:28.660677 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":365,"threads_started":1}
I20260619 16:54:28.665408 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:28.669809 3712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dafe6349cb4f9e95b5d2403a934a23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } }
I20260619 16:54:28.670321 3712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:28.670452 3712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83dafe6349cb4f9e95b5d2403a934a23, State: Initialized, Role: FOLLOWER
I20260619 16:54:28.671152 3712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dafe6349cb4f9e95b5d2403a934a23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } }
I20260619 16:54:28.671417 3712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:28.671571 3712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:28.671775 3712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:28.673997 3712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dafe6349cb4f9e95b5d2403a934a23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } }
I20260619 16:54:28.674553 3712 leader_election.cc:302] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83dafe6349cb4f9e95b5d2403a934a23; no voters:
I20260619 16:54:28.675307 3712 leader_election.cc:288] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:28.675433 3722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:28.676270 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:28.669670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:48891 (local address 127.0.0.1:44497)
0619 16:54:28.669872 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:28.669881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:28.670240 (+ 359us) server_negotiation.cc:407] Connection header received
0619 16:54:28.670460 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:28.670464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:28.670525 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:28.670628 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:28.671190 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:28.672130 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:28.673400 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:28.673852 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:28.673990 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:28.675259 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:28.675266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:28.675271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:28.675291 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:28.675363 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:28.675386 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:28.675516 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:28.675733 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 16:54:28.675808 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:54:28.677234 3722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [term 1 LEADER]: Becoming Leader. State: Replica: 83dafe6349cb4f9e95b5d2403a934a23, State: Running, Role: LEADER
I20260619 16:54:28.677878 3722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dafe6349cb4f9e95b5d2403a934a23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } }
I20260619 16:54:28.679926 3723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83dafe6349cb4f9e95b5d2403a934a23. Latest consensus state: current_term: 1 leader_uuid: "83dafe6349cb4f9e95b5d2403a934a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dafe6349cb4f9e95b5d2403a934a23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } } }
I20260619 16:54:28.680372 3723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:28.680683 3724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83dafe6349cb4f9e95b5d2403a934a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dafe6349cb4f9e95b5d2403a934a23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } } }
I20260619 16:54:28.681008 3724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:28.681313 3712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:28.683277 3726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:28.688508 3726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:28.698336 3726 catalog_manager.cc:1347] Generated new cluster ID: 82e4861b031548a09ac2ff347d66f1cf
I20260619 16:54:28.698457 3726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:28.710408 3726 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:28.711752 3726 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:28.720543 3726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83dafe6349cb4f9e95b5d2403a934a23: Generated new TSK 0
I20260619 16:54:28.721637 3726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:28.728157 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3645
W20260619 16:54:28.729853 3707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:46469
W20260619 16:54:28.730635 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3645: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:28.730803 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 3645
W20260619 16:54:28.747452 3630 connection.cc:535] client connection to 127.0.0.1:44497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44497): Connection reset by peer (error 104)
2026-06-19T16:54:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:28 dist-test-slave-qb36 krb5kdc[3743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:28 dist-test-slave-qb36 krb5kdc[3743](info): set up 2 sockets
Jun 19 16:54:28 dist-test-slave-qb36 krb5kdc[3743](info): commencing operation
krb5kdc: starting...
W20260619 16:54:29.843513 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:30 dist-test-slave-qb36 krb5kdc[3743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:30.107133 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:30.449352 3756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:30.449684 3756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:30.449780 3756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:30.457069 3756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:30.457195 3756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:30.457245 3756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:30.457307 3756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:30.457352 3756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:30.467347 3756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:30.468659 3756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:30.469949 3756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:30.481837 3764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:30.481884 3762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:30.482383 3761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:30.482575 3756 server_base.cc:1034] running on GCE node
I20260619 16:54:30.483459 3756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:30.484938 3756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:30.486189 3756 hybrid_clock.cc:648] HybridClock initialized: now 1781888070486129 us; error 43 us; skew 500 ppm
I20260619 16:54:30.486548 3756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:30 dist-test-slave-qb36 krb5kdc[3743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:30.493054 3756 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:30.494696 3756 webserver.cc:460] Webserver started at http://127.0.0.1:44993/ using document root <none> and password file <none>
I20260619 16:54:30.495510 3756 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:30.495685 3756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:30.496008 3756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:30.499137 3756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "af9c99b7b7264068bc37a5cd3066f842"
format_stamp: "Formatted at 2026-06-19 16:54:30 on dist-test-slave-qb36"
server_key: "85f85490456cd111feea72c8b913c7c2"
server_key_iv: "83cf9e4cd4c2ad0536d0b8f24e2c58cc"
server_key_version: "encryptionkey@0"
I20260619 16:54:30.499938 3756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "af9c99b7b7264068bc37a5cd3066f842"
format_stamp: "Formatted at 2026-06-19 16:54:30 on dist-test-slave-qb36"
server_key: "85f85490456cd111feea72c8b913c7c2"
server_key_iv: "83cf9e4cd4c2ad0536d0b8f24e2c58cc"
server_key_version: "encryptionkey@0"
I20260619 16:54:30.505301 3756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 16:54:30.508913 3771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:30.510293 3756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:54:30.510548 3756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "af9c99b7b7264068bc37a5cd3066f842"
format_stamp: "Formatted at 2026-06-19 16:54:30 on dist-test-slave-qb36"
server_key: "85f85490456cd111feea72c8b913c7c2"
server_key_iv: "83cf9e4cd4c2ad0536d0b8f24e2c58cc"
server_key_version: "encryptionkey@0"
I20260619 16:54:30.510778 3756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:30.516853 3756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:30.521080 3756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:30.521400 3756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:30.540120 3756 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35985
I20260619 16:54:30.540138 3822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35985 every 8 connection(s)
I20260619 16:54:30.541599 3756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:30.542594 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 3756
I20260619 16:54:30.542798 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:30.543325 14948 external_mini_cluster.cc:1428] Setting key afd27eba6f46fb3bd4c058e29339ede8
I20260619 16:54:30.546279 3823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:30.551250 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:30.545185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:48399 (local address 127.0.0.1:35985)
0619 16:54:30.545979 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:30.545992 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:30.546033 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:54:30.547807 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:30.547830 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:30.548161 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:30.548485 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:30.549549 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:30.549556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:30.549568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:30.549588 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:30.549654 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:30.549679 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:30.549843 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:30.549986 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 16:54:30.550185 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":355,"threads_started":1}
I20260619 16:54:30.555626 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:30.559119 3823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842: Bootstrap starting.
I20260619 16:54:30.563607 3823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:30.563762 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:30.561028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:37229 (local address 127.0.0.1:35985)
0619 16:54:30.561267 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:30.561276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:30.561716 (+ 440us) server_negotiation.cc:407] Connection header received
0619 16:54:30.561871 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:30.561876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:30.561940 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:30.562055 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:30.562988 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:30.562994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:30.563000 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:30.563018 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:30.563094 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:30.563115 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:30.563237 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:30.563306 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 16:54:30.563369 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:54:30.564908 3823 log.cc:824] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:30.568006 3823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842: No bootstrap required, opened a new log
I20260619 16:54:30.575938 3823 raft_consensus.cc:348] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9c99b7b7264068bc37a5cd3066f842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } }
I20260619 16:54:30.576387 3823 raft_consensus.cc:374] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:30.576499 3823 raft_consensus.cc:729] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9c99b7b7264068bc37a5cd3066f842, State: Initialized, Role: FOLLOWER
I20260619 16:54:30.577148 3823 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9c99b7b7264068bc37a5cd3066f842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } }
I20260619 16:54:30.577404 3823 raft_consensus.cc:388] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:30.577566 3823 raft_consensus.cc:482] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:30.577752 3823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:30.580443 3823 raft_consensus.cc:504] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9c99b7b7264068bc37a5cd3066f842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } }
I20260619 16:54:30.580993 3823 leader_election.cc:302] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af9c99b7b7264068bc37a5cd3066f842; no voters:
I20260619 16:54:30.581615 3823 leader_election.cc:288] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:30.581765 3833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:30.582504 3833 raft_consensus.cc:686] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [term 1 LEADER]: Becoming Leader. State: Replica: af9c99b7b7264068bc37a5cd3066f842, State: Running, Role: LEADER
I20260619 16:54:30.583173 3833 consensus_queue.cc:237] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9c99b7b7264068bc37a5cd3066f842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } }
I20260619 16:54:30.583959 3823 sys_catalog.cc:564] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:30.585919 3835 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af9c99b7b7264068bc37a5cd3066f842. Latest consensus state: current_term: 1 leader_uuid: "af9c99b7b7264068bc37a5cd3066f842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9c99b7b7264068bc37a5cd3066f842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } } }
I20260619 16:54:30.586421 3835 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:30.586570 3834 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af9c99b7b7264068bc37a5cd3066f842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9c99b7b7264068bc37a5cd3066f842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35985 } } }
I20260619 16:54:30.586889 3834 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:30.589010 3838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:30.595733 3838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:30.605041 3838 catalog_manager.cc:1347] Generated new cluster ID: c08261d3e4c84d53a2b1a10febbbef01
I20260619 16:54:30.605180 3838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:30.624503 3838 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:30.624971 3838 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:30.631309 3838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af9c99b7b7264068bc37a5cd3066f842: Generated new TSK 0
I20260619 16:54:30.632383 3838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:30.649650 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3756
W20260619 16:54:30.651558 3818 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:48399
W20260619 16:54:30.652328 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3756: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:30.652500 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 3756
2026-06-19T16:54:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:30 dist-test-slave-qb36 krb5kdc[3854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:30 dist-test-slave-qb36 krb5kdc[3854](info): set up 2 sockets
Jun 19 16:54:30 dist-test-slave-qb36 krb5kdc[3854](info): commencing operation
krb5kdc: starting...
W20260619 16:54:31.779634 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:31 dist-test-slave-qb36 krb5kdc[3854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:32.042127 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:32.372073 3867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:32.372454 3867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:32.372640 3867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:32.380056 3867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:32.380235 3867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:32.380349 3867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:32.380466 3867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:32.380553 3867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:32.390609 3867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:32.391989 3867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:32.393359 3867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:32.405336 3872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:32.405310 3873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:32.405380 3875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:32.406229 3867 server_base.cc:1034] running on GCE node
I20260619 16:54:32.407017 3867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:32.408519 3867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:32.409776 3867 hybrid_clock.cc:648] HybridClock initialized: now 1781888072409725 us; error 45 us; skew 500 ppm
I20260619 16:54:32.410132 3867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:32 dist-test-slave-qb36 krb5kdc[3854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:32.413268 3867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:32.414824 3867 webserver.cc:460] Webserver started at http://127.0.0.1:34527/ using document root <none> and password file <none>
I20260619 16:54:32.415730 3867 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:32.415876 3867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:32.416184 3867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:32.419461 3867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "c1538ff78a8b413b87d829d2d1315091"
format_stamp: "Formatted at 2026-06-19 16:54:32 on dist-test-slave-qb36"
server_key: "3559bed0427ec5078d179764448aa8b0"
server_key_iv: "272dc0df728f4f5b4c090600b3b2f3e8"
server_key_version: "encryptionkey@0"
I20260619 16:54:32.420306 3867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "c1538ff78a8b413b87d829d2d1315091"
format_stamp: "Formatted at 2026-06-19 16:54:32 on dist-test-slave-qb36"
server_key: "3559bed0427ec5078d179764448aa8b0"
server_key_iv: "272dc0df728f4f5b4c090600b3b2f3e8"
server_key_version: "encryptionkey@0"
I20260619 16:54:32.425575 3867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:54:32.429224 3882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:32.430610 3867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:54:32.430827 3867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "c1538ff78a8b413b87d829d2d1315091"
format_stamp: "Formatted at 2026-06-19 16:54:32 on dist-test-slave-qb36"
server_key: "3559bed0427ec5078d179764448aa8b0"
server_key_iv: "272dc0df728f4f5b4c090600b3b2f3e8"
server_key_version: "encryptionkey@0"
I20260619 16:54:32.431037 3867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:32.447260 3867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:32.451653 3867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:32.451944 3867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:32.470449 3867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39847
I20260619 16:54:32.470475 3933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39847 every 8 connection(s)
I20260619 16:54:32.471908 3867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:32.475946 3934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:32.477222 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 3867
I20260619 16:54:32.477404 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:32.477995 14948 external_mini_cluster.cc:1428] Setting key 1f7394fa6854ef2da73dbd4e6ea0829a
I20260619 16:54:32.489195 3934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091: Bootstrap starting.
Jun 19 16:54:32 dist-test-slave-qb36 krb5kdc[3854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:54:32.492398 3934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:32.493422 3934 log.cc:824] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:32.496621 3934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091: No bootstrap required, opened a new log
I20260619 16:54:32.500914 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:32.479999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:59899 (local address 127.0.0.1:39847)
0619 16:54:32.480792 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:32.480804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:32.480843 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:54:32.482805 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:32.482830 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:32.483321 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:32.483718 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:32.485215 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:32.486539 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:32.487957 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:32.488666 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:32.488869 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:32.493083 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:32.493090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:32.493105 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:32.493144 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:32.496644 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:32.497476 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:32.497483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:32.497488 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:32.497656 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:32.498145 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:32.498149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:32.498157 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:32.498685 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:32.499037 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:32.499590 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 16:54:32.499912 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":332,"threads_started":1}
I20260619 16:54:32.504963 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:32.505339 3934 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1538ff78a8b413b87d829d2d1315091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } }
I20260619 16:54:32.505695 3934 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:32.505800 3934 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1538ff78a8b413b87d829d2d1315091, State: Initialized, Role: FOLLOWER
I20260619 16:54:32.506382 3934 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1538ff78a8b413b87d829d2d1315091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } }
I20260619 16:54:32.506628 3934 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:32.506770 3934 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:32.506946 3934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:32.509330 3934 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1538ff78a8b413b87d829d2d1315091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } }
I20260619 16:54:32.509968 3934 leader_election.cc:302] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1538ff78a8b413b87d829d2d1315091; no voters:
I20260619 16:54:32.510752 3934 leader_election.cc:288] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:32.510869 3944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:32.513024 3934 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:32.515720 3944 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [term 1 LEADER]: Becoming Leader. State: Replica: c1538ff78a8b413b87d829d2d1315091, State: Running, Role: LEADER
I20260619 16:54:32.516418 3944 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1538ff78a8b413b87d829d2d1315091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } }
I20260619 16:54:32.524544 3945 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1538ff78a8b413b87d829d2d1315091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1538ff78a8b413b87d829d2d1315091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } } }
I20260619 16:54:32.524996 3945 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:32.525499 3946 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1538ff78a8b413b87d829d2d1315091. Latest consensus state: current_term: 1 leader_uuid: "c1538ff78a8b413b87d829d2d1315091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1538ff78a8b413b87d829d2d1315091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } } }
I20260619 16:54:32.525868 3946 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:32.531734 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:32.509541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:40261 (local address 127.0.0.1:39847)
0619 16:54:32.509786 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:32.509792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:32.510138 (+ 346us) server_negotiation.cc:407] Connection header received
0619 16:54:32.510310 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:32.510314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:32.510375 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:32.510514 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:32.512546 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:32.514177 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:32.516673 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:32.517419 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:32.517556 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:32.521870 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:32.521880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:32.521887 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:32.521945 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:32.526586 (+ 4641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:32.527897 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:32.527905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:32.527911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:32.528086 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:32.528795 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:32.528803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:32.528808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:32.529307 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:32.530730 (+ 1423us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:32.530958 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 16:54:32.531221 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 16:54:32.537649 3958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:54:32.537789 3958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:54:32.537959 3959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:32.540052 3959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:32.549217 3959 catalog_manager.cc:1347] Generated new cluster ID: d97a2ad1c6a24e689e3a07608fe11727
I20260619 16:54:32.549335 3959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:32.568017 3959 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:32.569341 3959 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:32.581496 3959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1538ff78a8b413b87d829d2d1315091: Generated new TSK 0
I20260619 16:54:32.582423 3959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:32.593811 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3867
I20260619 16:54:32.595758 3929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:32.688941 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 3867
W20260619 16:54:32.706283 3852 connection.cc:535] client connection to 127.0.0.1:39847 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39847): Connection reset by peer (error 104)
2026-06-19T16:54:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2037 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:32 dist-test-slave-qb36 krb5kdc[3966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:32 dist-test-slave-qb36 krb5kdc[3966](info): set up 2 sockets
Jun 19 16:54:32 dist-test-slave-qb36 krb5kdc[3966](info): commencing operation
krb5kdc: starting...
W20260619 16:54:33.807551 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:33 dist-test-slave-qb36 krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:34.072233 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:34.406625 3979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:34.406980 3979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:34.407155 3979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:34.414531 3979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:34.414716 3979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:34.414830 3979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:34.414940 3979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:34.415019 3979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:34.425676 3979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:34.427205 3979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:34.428505 3979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:34.440409 3984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:34.440466 3987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:34.440483 3985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:34.441345 3979 server_base.cc:1034] running on GCE node
I20260619 16:54:34.442171 3979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:34.443704 3979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:34.444943 3979 hybrid_clock.cc:648] HybridClock initialized: now 1781888074444898 us; error 37 us; skew 500 ppm
I20260619 16:54:34.445346 3979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:34 dist-test-slave-qb36 krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:34.448602 3979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:34.450157 3979 webserver.cc:460] Webserver started at http://127.0.0.1:45135/ using document root <none> and password file <none>
I20260619 16:54:34.450855 3979 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:34.450990 3979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:34.451392 3979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:34.454593 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "2778e6494cfa419b9457581af1ce0ca3"
format_stamp: "Formatted at 2026-06-19 16:54:34 on dist-test-slave-qb36"
server_key: "8a814c0277e620eb6adc628eb95f1f73"
server_key_iv: "c268da3788ef3ab8e1490ab25284a5bf"
server_key_version: "encryptionkey@0"
I20260619 16:54:34.455571 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "2778e6494cfa419b9457581af1ce0ca3"
format_stamp: "Formatted at 2026-06-19 16:54:34 on dist-test-slave-qb36"
server_key: "8a814c0277e620eb6adc628eb95f1f73"
server_key_iv: "c268da3788ef3ab8e1490ab25284a5bf"
server_key_version: "encryptionkey@0"
I20260619 16:54:34.460829 3979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 16:54:34.464363 3994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:34.465673 3979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:34.465878 3979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "2778e6494cfa419b9457581af1ce0ca3"
format_stamp: "Formatted at 2026-06-19 16:54:34 on dist-test-slave-qb36"
server_key: "8a814c0277e620eb6adc628eb95f1f73"
server_key_iv: "c268da3788ef3ab8e1490ab25284a5bf"
server_key_version: "encryptionkey@0"
I20260619 16:54:34.466081 3979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:34.491009 3979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:34.495729 3979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:34.496044 3979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:34.514822 4045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41419 every 8 connection(s)
I20260619 16:54:34.514839 3979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41419
I20260619 16:54:34.516379 3979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:34.518280 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 3979
I20260619 16:54:34.518457 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:34.519021 14948 external_mini_cluster.cc:1428] Setting key a0ab66285dcc0ac140f648a493753559
I20260619 16:54:34.520812 4046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:34.532153 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3: Bootstrap starting.
I20260619 16:54:34.532405 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:34.520922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:38065 (local address 127.0.0.1:41419)
0619 16:54:34.521667 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:34.521679 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:34.521723 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:54:34.523483 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:34.523516 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:34.523957 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:34.524392 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:34.524964 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:34.526424 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:34.527898 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:34.528587 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:34.528784 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:34.530080 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:34.530087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:34.530100 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:34.530646 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:34.530721 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:34.530747 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:34.530917 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:34.531236 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:54:34.531404 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":318,"threads_started":1}
I20260619 16:54:34.535526 4046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:34.536252 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:34.536695 4046 log.cc:824] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:34.539418 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3: No bootstrap required, opened a new log
I20260619 16:54:34.547564 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:34.540753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:34317 (local address 127.0.0.1:41419)
0619 16:54:34.540975 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:34.540983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:34.541308 (+ 325us) server_negotiation.cc:407] Connection header received
0619 16:54:34.541487 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:34.541490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:34.541549 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:34.541634 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:34.542237 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:34.543085 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:34.544975 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:34.545464 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:34.545564 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:34.546741 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:34.546745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:34.546748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:34.546766 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:34.546817 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:34.546831 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:34.546915 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:34.547157 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 16:54:34.547200 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:54:34.549340 4046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2778e6494cfa419b9457581af1ce0ca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41419 } }
I20260619 16:54:34.549728 4046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:34.549842 4046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2778e6494cfa419b9457581af1ce0ca3, State: Initialized, Role: FOLLOWER
I20260619 16:54:34.550405 4046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2778e6494cfa419b9457581af1ce0ca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41419 } }
I20260619 16:54:34.550652 4046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:34.550798 4046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:34.550979 4046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:34.553467 4046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2778e6494cfa419b9457581af1ce0ca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41419 } }
I20260619 16:54:34.554010 4046 leader_election.cc:302] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2778e6494cfa419b9457581af1ce0ca3; no voters:
I20260619 16:54:34.554672 4046 leader_election.cc:288] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:34.554795 4056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:34.555668 4056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [term 1 LEADER]: Becoming Leader. State: Replica: 2778e6494cfa419b9457581af1ce0ca3, State: Running, Role: LEADER
I20260619 16:54:34.556226 4056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2778e6494cfa419b9457581af1ce0ca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41419 } }
I20260619 16:54:34.557199 4046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:34.559319 4058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2778e6494cfa419b9457581af1ce0ca3. Latest consensus state: current_term: 1 leader_uuid: "2778e6494cfa419b9457581af1ce0ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2778e6494cfa419b9457581af1ce0ca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41419 } } }
I20260619 16:54:34.559273 4057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2778e6494cfa419b9457581af1ce0ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2778e6494cfa419b9457581af1ce0ca3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41419 } } }
I20260619 16:54:34.559702 4058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:34.559751 4057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:34.564388 4062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:34.570046 4062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:34.579800 4062 catalog_manager.cc:1347] Generated new cluster ID: ca3a8987031c40eaace35b30d3d3247f
I20260619 16:54:34.579937 4062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:34.606976 4062 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:34.608361 4062 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:34.621802 4062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2778e6494cfa419b9457581af1ce0ca3: Generated new TSK 0
I20260619 16:54:34.622711 4062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:34.634714 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3979
W20260619 16:54:34.636475 4041 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:38065
W20260619 16:54:34.637307 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 3979: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:34.637509 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 3979
W20260619 16:54:34.654695 3964 connection.cc:535] client connection to 127.0.0.1:41419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41419): Connection reset by peer (error 104)
2026-06-19T16:54:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:34 dist-test-slave-qb36 krb5kdc[4077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:34 dist-test-slave-qb36 krb5kdc[4077](info): set up 2 sockets
Jun 19 16:54:34 dist-test-slave-qb36 krb5kdc[4077](info): commencing operation
krb5kdc: starting...
W20260619 16:54:35.747589 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:35 dist-test-slave-qb36 krb5kdc[4077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:54:36.008701 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:36.343766 4090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:36.344081 4090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:36.344259 4090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:36.351584 4090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:36.351725 4090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:36.351822 4090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:36.351940 4090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:36.352028 4090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:36.362535 4090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:36.363991 4090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:36.365368 4090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:36.377621 4098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:36.377702 4095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:36.377702 4096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:36.378453 4090 server_base.cc:1034] running on GCE node
I20260619 16:54:36.379349 4090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:36.380788 4090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:36.382050 4090 hybrid_clock.cc:648] HybridClock initialized: now 1781888076382003 us; error 48 us; skew 500 ppm
I20260619 16:54:36.382431 4090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:36 dist-test-slave-qb36 krb5kdc[4077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:36.390636 4090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:54:36.392493 4090 webserver.cc:460] Webserver started at http://127.0.0.1:39999/ using document root <none> and password file <none>
I20260619 16:54:36.393244 4090 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:36.393399 4090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:36.393723 4090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:36.396925 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "57e746dde80149e6b40b59ccd9e95f1a"
format_stamp: "Formatted at 2026-06-19 16:54:36 on dist-test-slave-qb36"
server_key: "ded452c4482c6083eeed38511dda6134"
server_key_iv: "b674386808bb7621e6b5a2c80e31ae65"
server_key_version: "encryptionkey@0"
I20260619 16:54:36.397728 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "57e746dde80149e6b40b59ccd9e95f1a"
format_stamp: "Formatted at 2026-06-19 16:54:36 on dist-test-slave-qb36"
server_key: "ded452c4482c6083eeed38511dda6134"
server_key_iv: "b674386808bb7621e6b5a2c80e31ae65"
server_key_version: "encryptionkey@0"
I20260619 16:54:36.403220 4090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 16:54:36.406927 4105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:36.408486 4090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:36.408715 4090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "57e746dde80149e6b40b59ccd9e95f1a"
format_stamp: "Formatted at 2026-06-19 16:54:36 on dist-test-slave-qb36"
server_key: "ded452c4482c6083eeed38511dda6134"
server_key_iv: "b674386808bb7621e6b5a2c80e31ae65"
server_key_version: "encryptionkey@0"
I20260619 16:54:36.408943 4090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:36.414954 4090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:36.419761 4090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:36.420090 4090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:36.439373 4090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45225
I20260619 16:54:36.439424 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45225 every 8 connection(s)
I20260619 16:54:36.440889 4090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:36.443775 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 4090
I20260619 16:54:36.443959 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:36.444442 14948 external_mini_cluster.cc:1428] Setting key f4fe78ee62064aa9c4c7127b37f04b1e
I20260619 16:54:36.445225 4157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:36.453225 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:36.446401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:45357 (local address 127.0.0.1:45225)
0619 16:54:36.447246 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:36.447264 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:36.447317 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:54:36.449050 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:36.449080 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:36.449576 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:36.450053 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:36.451106 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:36.451117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:36.451143 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:36.451169 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:36.451259 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:36.451292 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:36.451517 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:36.451653 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:54:36.451873 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":329,"threads_started":1}
I20260619 16:54:36.457396 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:36.460619 4157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a: Bootstrap starting.
I20260619 16:54:36.464470 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:36.461769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:48957 (local address 127.0.0.1:45225)
0619 16:54:36.461997 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:36.462007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:36.462483 (+ 476us) server_negotiation.cc:407] Connection header received
0619 16:54:36.462664 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:36.462667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:36.462718 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:36.462806 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:36.463794 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:36.463798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:36.463801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:36.463816 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:36.463854 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:36.463869 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:36.463963 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:36.464115 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 16:54:36.464164 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:54:36.464774 4157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:36.466161 4157 log.cc:824] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:36.469084 4157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a: No bootstrap required, opened a new log
I20260619 16:54:36.477551 4157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e746dde80149e6b40b59ccd9e95f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260619 16:54:36.477990 4157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:36.478101 4157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e746dde80149e6b40b59ccd9e95f1a, State: Initialized, Role: FOLLOWER
I20260619 16:54:36.478790 4157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e746dde80149e6b40b59ccd9e95f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260619 16:54:36.479106 4157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:36.479288 4157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:36.479472 4157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:36.481730 4157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e746dde80149e6b40b59ccd9e95f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260619 16:54:36.482172 4157 leader_election.cc:302] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57e746dde80149e6b40b59ccd9e95f1a; no voters:
I20260619 16:54:36.482687 4157 leader_election.cc:288] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:36.482851 4167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:36.483600 4167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [term 1 LEADER]: Becoming Leader. State: Replica: 57e746dde80149e6b40b59ccd9e95f1a, State: Running, Role: LEADER
I20260619 16:54:36.484164 4167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e746dde80149e6b40b59ccd9e95f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260619 16:54:36.484809 4157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:36.487283 4169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57e746dde80149e6b40b59ccd9e95f1a. Latest consensus state: current_term: 1 leader_uuid: "57e746dde80149e6b40b59ccd9e95f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e746dde80149e6b40b59ccd9e95f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } } }
I20260619 16:54:36.487741 4169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:36.489113 4176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:36.490358 4168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57e746dde80149e6b40b59ccd9e95f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e746dde80149e6b40b59ccd9e95f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } } }
I20260619 16:54:36.490660 4168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:36.502503 4176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:36.511644 4176 catalog_manager.cc:1347] Generated new cluster ID: a76f0b139aea4fe896cf899869b01d76
I20260619 16:54:36.511770 4176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:36.554827 4176 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:36.555346 4176 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:36.564679 4176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57e746dde80149e6b40b59ccd9e95f1a: Generated new TSK 0
I20260619 16:54:36.565553 4176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:36.578920 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4090
W20260619 16:54:36.580140 4152 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:45357
W20260619 16:54:36.580960 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4090: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:36.581135 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 4090
2026-06-19T16:54:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:36 dist-test-slave-qb36 krb5kdc[4188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:36 dist-test-slave-qb36 krb5kdc[4188](info): set up 2 sockets
Jun 19 16:54:36 dist-test-slave-qb36 krb5kdc[4188](info): commencing operation
krb5kdc: starting...
W20260619 16:54:37.695627 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:37 dist-test-slave-qb36 krb5kdc[4188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:37.959831 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35783
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:35783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:38.298678 4201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:38.299072 4201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:38.299237 4201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:38.306511 4201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:38.306668 4201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:38.306788 4201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:38.306888 4201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:38.306972 4201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:38.317116 4201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:35783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:35783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:38.318481 4201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:38.319864 4201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:38.331516 4206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:38.331449 4207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:38.331581 4209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:38.332726 4201 server_base.cc:1034] running on GCE node
I20260619 16:54:38.333534 4201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:38.334995 4201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:38.336266 4201 hybrid_clock.cc:648] HybridClock initialized: now 1781888078336209 us; error 51 us; skew 500 ppm
I20260619 16:54:38.336664 4201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:38 dist-test-slave-qb36 krb5kdc[4188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888078, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:38.339941 4201 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:38.341430 4201 webserver.cc:460] Webserver started at http://127.14.153.62:39853/ using document root <none> and password file <none>
I20260619 16:54:38.342070 4201 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:38.342178 4201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:38.342423 4201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:38.345486 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "692cfcb2ec034288bb2f705dd4f1124b"
format_stamp: "Formatted at 2026-06-19 16:54:38 on dist-test-slave-qb36"
server_key: "1f69a26b388ea4aae83088c894b26ee1"
server_key_iv: "4f46f70f05f29312f4da23f9b8ac9573"
server_key_version: "encryptionkey@0"
I20260619 16:54:38.346207 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "692cfcb2ec034288bb2f705dd4f1124b"
format_stamp: "Formatted at 2026-06-19 16:54:38 on dist-test-slave-qb36"
server_key: "1f69a26b388ea4aae83088c894b26ee1"
server_key_iv: "4f46f70f05f29312f4da23f9b8ac9573"
server_key_version: "encryptionkey@0"
I20260619 16:54:38.351392 4201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:54:38.354902 4216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:38.356410 4201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:38.356559 4201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "692cfcb2ec034288bb2f705dd4f1124b"
format_stamp: "Formatted at 2026-06-19 16:54:38 on dist-test-slave-qb36"
server_key: "1f69a26b388ea4aae83088c894b26ee1"
server_key_iv: "4f46f70f05f29312f4da23f9b8ac9573"
server_key_version: "encryptionkey@0"
I20260619 16:54:38.356740 4201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:38.380867 4201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:38.385253 4201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:38.385493 4201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:38.404239 4201 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:35783
I20260619 16:54:38.404263 4267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:35783 every 8 connection(s)
I20260619 16:54:38.405684 4201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:38.409554 4268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:38.415478 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 4201
I20260619 16:54:38.415693 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:38.416253 14948 external_mini_cluster.cc:1428] Setting key 3543884112a48e80c21aa2e2be9844cb
I20260619 16:54:38.421260 4268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b: Bootstrap starting.
I20260619 16:54:38.424443 4268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:38.425494 4268 log.cc:824] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:38.428613 4268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b: No bootstrap required, opened a new log
Jun 19 16:54:38 dist-test-slave-qb36 krb5kdc[4188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:38.439250 4268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692cfcb2ec034288bb2f705dd4f1124b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35783 } }
I20260619 16:54:38.439733 4268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:38.439839 4268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692cfcb2ec034288bb2f705dd4f1124b, State: Initialized, Role: FOLLOWER
I20260619 16:54:38.440523 4268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692cfcb2ec034288bb2f705dd4f1124b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35783 } }
I20260619 16:54:38.440828 4268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:38.440975 4268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:38.441092 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:38.418182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58133 (local address 127.14.153.62:35783)
0619 16:54:38.419166 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:38.419183 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:38.419225 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:54:38.420938 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:38.420963 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:38.421450 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:38.421867 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:38.423491 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:38.424860 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:38.426251 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:38.427154 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:38.427358 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:38.431794 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:38.431805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:38.431826 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:38.431882 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:38.436204 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:38.437043 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:38.437058 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:38.437068 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:38.437312 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:38.437776 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:38.437782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:38.437791 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:38.438511 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:38.438843 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:38.439292 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 16:54:38.439641 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":392,"threads_started":1}
I20260619 16:54:38.441190 4268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:38.443696 4268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692cfcb2ec034288bb2f705dd4f1124b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35783 } }
I20260619 16:54:38.444257 4268 leader_election.cc:302] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692cfcb2ec034288bb2f705dd4f1124b; no voters:
I20260619 16:54:38.444854 4268 leader_election.cc:288] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:38.445065 4273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:38.445319 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:38.445709 4273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [term 1 LEADER]: Becoming Leader. State: Replica: 692cfcb2ec034288bb2f705dd4f1124b, State: Running, Role: LEADER
I20260619 16:54:38.446326 4273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692cfcb2ec034288bb2f705dd4f1124b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35783 } }
I20260619 16:54:38.448180 4274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 692cfcb2ec034288bb2f705dd4f1124b. Latest consensus state: current_term: 1 leader_uuid: "692cfcb2ec034288bb2f705dd4f1124b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692cfcb2ec034288bb2f705dd4f1124b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35783 } } }
I20260619 16:54:38.448604 4274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:38.449005 4275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "692cfcb2ec034288bb2f705dd4f1124b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692cfcb2ec034288bb2f705dd4f1124b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 35783 } } }
I20260619 16:54:38.449251 4268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:38.449335 4275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:38.451548 4278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:38.464900 4278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:38.487425 4278 catalog_manager.cc:1347] Generated new cluster ID: 87553b98998141699a7a9316202056db
I20260619 16:54:38.487586 4278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:38.497907 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:38.464230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59857 (local address 127.14.153.62:35783)
0619 16:54:38.464427 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:38.464434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:38.472970 (+ 8536us) server_negotiation.cc:407] Connection header received
0619 16:54:38.473572 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:38.473582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:38.473689 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:38.473781 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:38.476781 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:38.478412 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:38.481589 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:38.482561 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:38.482691 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:38.489483 (+ 6792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:38.489491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:38.489497 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:38.489543 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:38.493471 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:38.494650 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:38.494656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:38.494660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:38.494832 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:38.495455 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:38.495460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:38.495464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:38.495876 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:38.496143 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:38.497298 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0619 16:54:38.497485 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:54:38.510689 4278 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:38.512614 4278 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:38.522825 4278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 692cfcb2ec034288bb2f705dd4f1124b: Generated new TSK 0
I20260619 16:54:38.523949 4278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:38.532720 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4201
I20260619 16:54:38.533957 4263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:38.627252 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 4201
W20260619 16:54:38.644640 4186 connection.cc:535] client connection to 127.14.153.62:35783 recv error: Network error: failed to read from TLS socket (remote: 127.14.153.62:35783): Connection reset by peer (error 104)
2026-06-19T16:54:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:38 dist-test-slave-qb36 krb5kdc[4300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:38 dist-test-slave-qb36 krb5kdc[4300](info): set up 2 sockets
Jun 19 16:54:38 dist-test-slave-qb36 krb5kdc[4300](info): commencing operation
krb5kdc: starting...
W20260619 16:54:39.739687 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:39 dist-test-slave-qb36 krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:40.002532 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:44441
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:44585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:44441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:40.338424 4313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:40.338804 4313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:40.338976 4313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:40.346483 4313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:40.346670 4313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:40.346791 4313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:40.346900 4313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:40.346987 4313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:40.357257 4313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:44585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:44441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:44441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:40.358726 4313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:40.360189 4313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:40.372154 4318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:40.372141 4319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:40.372180 4321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:40.372970 4313 server_base.cc:1034] running on GCE node
I20260619 16:54:40.373816 4313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:40.375388 4313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:40.376602 4313 hybrid_clock.cc:648] HybridClock initialized: now 1781888080376558 us; error 44 us; skew 500 ppm
I20260619 16:54:40.377033 4313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:40 dist-test-slave-qb36 krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:40.380295 4313 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:40.381852 4313 webserver.cc:460] Webserver started at http://127.14.153.62:34029/ using document root <none> and password file <none>
I20260619 16:54:40.382563 4313 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:40.382717 4313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:40.383071 4313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:40.386176 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "66bdd975908648018401978285b4615b"
format_stamp: "Formatted at 2026-06-19 16:54:40 on dist-test-slave-qb36"
server_key: "4a32f3600c3cfcbb40ce0ffcfac68ad1"
server_key_iv: "d5ee4a512003607cce7bc76dfae75f65"
server_key_version: "encryptionkey@0"
I20260619 16:54:40.386965 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "66bdd975908648018401978285b4615b"
format_stamp: "Formatted at 2026-06-19 16:54:40 on dist-test-slave-qb36"
server_key: "4a32f3600c3cfcbb40ce0ffcfac68ad1"
server_key_iv: "d5ee4a512003607cce7bc76dfae75f65"
server_key_version: "encryptionkey@0"
I20260619 16:54:40.392300 4313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:54:40.395848 4328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:40.397186 4313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:40.397374 4313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "66bdd975908648018401978285b4615b"
format_stamp: "Formatted at 2026-06-19 16:54:40 on dist-test-slave-qb36"
server_key: "4a32f3600c3cfcbb40ce0ffcfac68ad1"
server_key_iv: "d5ee4a512003607cce7bc76dfae75f65"
server_key_version: "encryptionkey@0"
I20260619 16:54:40.397576 4313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:40.415263 4313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:40.420035 4313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:40.420344 4313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:40.439538 4313 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:44441
I20260619 16:54:40.439538 4379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:44441 every 8 connection(s)
I20260619 16:54:40.441015 4313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:40.444849 4380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:40.447805 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 4313
I20260619 16:54:40.447981 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:40.448530 14948 external_mini_cluster.cc:1428] Setting key 6018d94a2616d6916ae425d6d0eca0fb
I20260619 16:54:40.457222 4380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b: Bootstrap starting.
I20260619 16:54:40.460956 4380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:40.462069 4380 log.cc:824] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:40.462405 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:40.450392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33727 (local address 127.14.153.62:44441)
0619 16:54:40.451218 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:40.451230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:40.451273 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:54:40.452974 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:40.452999 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:40.453411 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:40.453752 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:40.454312 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:40.455777 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:40.457644 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:40.458335 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:40.458480 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:40.459852 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:40.459860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:40.459874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:40.460421 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:40.460502 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:40.460526 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:40.460666 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:40.460959 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 16:54:40.461257 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":386,"threads_started":1}
I20260619 16:54:40.465380 4380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b: No bootstrap required, opened a new log
I20260619 16:54:40.466593 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:40.475638 4380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bdd975908648018401978285b4615b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44441 } }
I20260619 16:54:40.476135 4380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:40.476270 4380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bdd975908648018401978285b4615b, State: Initialized, Role: FOLLOWER
I20260619 16:54:40.476899 4380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bdd975908648018401978285b4615b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44441 } }
I20260619 16:54:40.477125 4380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:40.477262 4380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:40.477442 4380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:40.478524 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:40.471713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37491 (local address 127.14.153.62:44441)
0619 16:54:40.471899 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:40.471905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:40.472390 (+ 485us) server_negotiation.cc:407] Connection header received
0619 16:54:40.472545 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:40.472551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:40.472627 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:40.472747 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:40.473162 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:40.474377 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:40.475810 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:40.476372 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:40.476517 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:40.477703 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:40.477714 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:40.477719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:40.477746 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:40.477791 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:40.477816 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:40.477939 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:40.478085 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 16:54:40.478142 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:54:40.480464 4380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bdd975908648018401978285b4615b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44441 } }
I20260619 16:54:40.480903 4380 leader_election.cc:302] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66bdd975908648018401978285b4615b; no voters:
I20260619 16:54:40.481372 4380 leader_election.cc:288] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:40.481523 4390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:40.482236 4390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [term 1 LEADER]: Becoming Leader. State: Replica: 66bdd975908648018401978285b4615b, State: Running, Role: LEADER
I20260619 16:54:40.482789 4390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bdd975908648018401978285b4615b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44441 } }
I20260619 16:54:40.483412 4380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:40.486270 4391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66bdd975908648018401978285b4615b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bdd975908648018401978285b4615b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44441 } } }
I20260619 16:54:40.486233 4392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66bdd975908648018401978285b4615b. Latest consensus state: current_term: 1 leader_uuid: "66bdd975908648018401978285b4615b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bdd975908648018401978285b4615b" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44441 } } }
I20260619 16:54:40.486686 4391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:40.486686 4392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:40.489852 4399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:40.495333 4399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:40.504557 4399 catalog_manager.cc:1347] Generated new cluster ID: 3bec9bba0f1445689c119b78124e4c0d
I20260619 16:54:40.504788 4399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:40.536517 4399 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:40.538424 4399 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:40.548617 4399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66bdd975908648018401978285b4615b: Generated new TSK 0
I20260619 16:54:40.549681 4399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:40.558513 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4313
W20260619 16:54:40.560063 4375 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33727
W20260619 16:54:40.560860 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4313: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:40.561022 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 4313
W20260619 16:54:40.578141 4298 connection.cc:535] client connection to 127.14.153.62:44441 recv error: Network error: failed to read from TLS socket (remote: 127.14.153.62:44441): Connection reset by peer (error 104)
2026-06-19T16:54:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:40 dist-test-slave-qb36 krb5kdc[4411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:40 dist-test-slave-qb36 krb5kdc[4411](info): set up 2 sockets
Jun 19 16:54:40 dist-test-slave-qb36 krb5kdc[4411](info): commencing operation
krb5kdc: starting...
W20260619 16:54:41.683638 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:41 dist-test-slave-qb36 krb5kdc[4411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:41.950651 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:40929
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:40929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:42.282117 4424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:42.282421 4424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:42.282588 4424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:42.290076 4424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:42.290213 4424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:42.290324 4424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:42.290432 4424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:42.290512 4424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:42.301077 4424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:40929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:40929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:42.302455 4424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:42.303829 4424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:42.315635 4429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:42.315654 4430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:42.316313 4432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:42.316660 4424 server_base.cc:1034] running on GCE node
I20260619 16:54:42.317494 4424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:42.319000 4424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:42.320302 4424 hybrid_clock.cc:648] HybridClock initialized: now 1781888082320252 us; error 40 us; skew 500 ppm
I20260619 16:54:42.320749 4424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:42 dist-test-slave-qb36 krb5kdc[4411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:42.323930 4424 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:42.325510 4424 webserver.cc:460] Webserver started at http://127.14.153.62:42595/ using document root <none> and password file <none>
I20260619 16:54:42.326242 4424 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:42.326390 4424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:42.326715 4424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:42.329849 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "95b2d9593461409b8e4045b6fcbdda89"
format_stamp: "Formatted at 2026-06-19 16:54:42 on dist-test-slave-qb36"
server_key: "f99e6d1b2f72da9473077327d3dfd59f"
server_key_iv: "1fed83f00162efbc4346c2a4808abdb6"
server_key_version: "encryptionkey@0"
I20260619 16:54:42.330655 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "95b2d9593461409b8e4045b6fcbdda89"
format_stamp: "Formatted at 2026-06-19 16:54:42 on dist-test-slave-qb36"
server_key: "f99e6d1b2f72da9473077327d3dfd59f"
server_key_iv: "1fed83f00162efbc4346c2a4808abdb6"
server_key_version: "encryptionkey@0"
I20260619 16:54:42.336146 4424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 16:54:42.339870 4439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:42.341382 4424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:42.341580 4424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "95b2d9593461409b8e4045b6fcbdda89"
format_stamp: "Formatted at 2026-06-19 16:54:42 on dist-test-slave-qb36"
server_key: "f99e6d1b2f72da9473077327d3dfd59f"
server_key_iv: "1fed83f00162efbc4346c2a4808abdb6"
server_key_version: "encryptionkey@0"
I20260619 16:54:42.341814 4424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:42.347831 4424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:42.352263 4424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:42.352579 4424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:42.371769 4424 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:40929
I20260619 16:54:42.371800 4490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:40929 every 8 connection(s)
I20260619 16:54:42.373251 4424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:42.376312 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 4424
I20260619 16:54:42.376507 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:42.376947 14948 external_mini_cluster.cc:1428] Setting key d3b447310558f0be592d590df9f5ffb5
I20260619 16:54:42.377224 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:42.386178 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:42.378773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34415 (local address 127.14.153.62:40929)
0619 16:54:42.379652 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:42.379668 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:42.379722 (+ 54us) server_negotiation.cc:407] Connection header received
0619 16:54:42.382105 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:42.382137 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:42.382605 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:42.383101 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:42.384185 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:42.384194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:42.384211 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:42.384229 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:42.384305 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:42.384338 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:42.384567 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:42.384720 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 16:54:42.384914 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":385,"threads_started":1}
I20260619 16:54:42.390015 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:42.391882 4491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89: Bootstrap starting.
I20260619 16:54:42.395430 4491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:42.396662 4491 log.cc:824] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:42.397132 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:42.394276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55059 (local address 127.14.153.62:40929)
0619 16:54:42.394450 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:42.394457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:42.394998 (+ 541us) server_negotiation.cc:407] Connection header received
0619 16:54:42.395204 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:42.395213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:42.395283 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:42.395405 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:42.396494 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:42.396499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:42.396503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:42.396519 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:42.396564 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:42.396581 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:42.396723 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:42.396781 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 16:54:42.396839 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:54:42.399935 4491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89: No bootstrap required, opened a new log
I20260619 16:54:42.407482 4491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b2d9593461409b8e4045b6fcbdda89" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40929 } }
I20260619 16:54:42.407894 4491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:42.408012 4491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b2d9593461409b8e4045b6fcbdda89, State: Initialized, Role: FOLLOWER
I20260619 16:54:42.408601 4491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b2d9593461409b8e4045b6fcbdda89" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40929 } }
I20260619 16:54:42.408870 4491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:42.409026 4491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:42.409245 4491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:42.411579 4491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b2d9593461409b8e4045b6fcbdda89" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40929 } }
I20260619 16:54:42.412091 4491 leader_election.cc:302] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95b2d9593461409b8e4045b6fcbdda89; no voters:
I20260619 16:54:42.412675 4491 leader_election.cc:288] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:42.412849 4501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:42.413568 4501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [term 1 LEADER]: Becoming Leader. State: Replica: 95b2d9593461409b8e4045b6fcbdda89, State: Running, Role: LEADER
I20260619 16:54:42.414103 4501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b2d9593461409b8e4045b6fcbdda89" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40929 } }
I20260619 16:54:42.414853 4491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:42.417344 4503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95b2d9593461409b8e4045b6fcbdda89. Latest consensus state: current_term: 1 leader_uuid: "95b2d9593461409b8e4045b6fcbdda89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b2d9593461409b8e4045b6fcbdda89" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40929 } } }
I20260619 16:54:42.417413 4502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95b2d9593461409b8e4045b6fcbdda89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b2d9593461409b8e4045b6fcbdda89" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40929 } } }
I20260619 16:54:42.417913 4503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:42.417913 4502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:42.418896 4510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:42.424674 4510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:42.434455 4510 catalog_manager.cc:1347] Generated new cluster ID: be26f106348442b1b4ee01094a12c9a8
I20260619 16:54:42.434588 4510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:42.451797 4510 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:42.452378 4510 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:42.461230 4510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95b2d9593461409b8e4045b6fcbdda89: Generated new TSK 0
I20260619 16:54:42.462301 4510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:42.476028 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4424
W20260619 16:54:42.477175 4486 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34415
W20260619 16:54:42.477893 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4424: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:42.478051 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 4424
2026-06-19T16:54:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1916 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:42 dist-test-slave-qb36 krb5kdc[4522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:42 dist-test-slave-qb36 krb5kdc[4522](info): set up 2 sockets
Jun 19 16:54:42 dist-test-slave-qb36 krb5kdc[4522](info): commencing operation
krb5kdc: starting...
W20260619 16:54:43.599637 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:43 dist-test-slave-qb36 krb5kdc[4522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:43.866380 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:44227
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:44227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:44.202031 4535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:44.202339 4535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:44.202431 4535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:44.209729 4535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:44.209853 4535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:44.209913 4535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:44.209973 4535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:44.210019 4535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:44.220041 4535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:44227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:44227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:44.221379 4535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:44.222695 4535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:44.234721 4540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:44.234750 4541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:44.234908 4543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:44.235581 4535 server_base.cc:1034] running on GCE node
I20260619 16:54:44.236410 4535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:44.237890 4535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:44.239166 4535 hybrid_clock.cc:648] HybridClock initialized: now 1781888084239111 us; error 40 us; skew 500 ppm
I20260619 16:54:44.239558 4535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:44 dist-test-slave-qb36 krb5kdc[4522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:44.242851 4535 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:44.244432 4535 webserver.cc:460] Webserver started at http://127.14.153.62:34137/ using document root <none> and password file <none>
I20260619 16:54:44.245160 4535 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:44.245314 4535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:44.245627 4535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:44.248860 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "a44f13d40f8e43e09533f077b3d5df2c"
format_stamp: "Formatted at 2026-06-19 16:54:44 on dist-test-slave-qb36"
server_key: "600bf9987be3343c3f2988d27c1c289b"
server_key_iv: "8a9a6f918f04bf4e9036f65fdc539c0e"
server_key_version: "encryptionkey@0"
I20260619 16:54:44.249643 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "a44f13d40f8e43e09533f077b3d5df2c"
format_stamp: "Formatted at 2026-06-19 16:54:44 on dist-test-slave-qb36"
server_key: "600bf9987be3343c3f2988d27c1c289b"
server_key_iv: "8a9a6f918f04bf4e9036f65fdc539c0e"
server_key_version: "encryptionkey@0"
I20260619 16:54:44.255244 4535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 16:54:44.258919 4550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:44.260450 4535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:44.260641 4535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "a44f13d40f8e43e09533f077b3d5df2c"
format_stamp: "Formatted at 2026-06-19 16:54:44 on dist-test-slave-qb36"
server_key: "600bf9987be3343c3f2988d27c1c289b"
server_key_iv: "8a9a6f918f04bf4e9036f65fdc539c0e"
server_key_version: "encryptionkey@0"
I20260619 16:54:44.260843 4535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:44.277645 4535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:44.281805 4535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:44.282101 4535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:44.301251 4535 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:44227
I20260619 16:54:44.301255 4601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:44227 every 8 connection(s)
I20260619 16:54:44.302661 4535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:44.306348 4602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:44.311403 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 4535
I20260619 16:54:44.311579 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:44.312052 14948 external_mini_cluster.cc:1428] Setting key 4a21d3b251c91e161503a2f8563602b1
I20260619 16:54:44.317556 4602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c: Bootstrap starting.
I20260619 16:54:44.320822 4602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:44.321846 4602 log.cc:824] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:44.325460 4602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c: No bootstrap required, opened a new log
Jun 19 16:54:44 dist-test-slave-qb36 krb5kdc[4522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:44.336382 4602 raft_consensus.cc:348] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f13d40f8e43e09533f077b3d5df2c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44227 } }
I20260619 16:54:44.336848 4602 raft_consensus.cc:374] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:44.336966 4602 raft_consensus.cc:729] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f13d40f8e43e09533f077b3d5df2c, State: Initialized, Role: FOLLOWER
I20260619 16:54:44.337054 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:44.313950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53573 (local address 127.14.153.62:44227)
0619 16:54:44.314863 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:44.314875 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:44.314921 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:54:44.316765 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:44.316790 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:44.317241 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:44.317659 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:44.319159 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:44.320479 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:44.321836 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:44.322877 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:44.323130 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:44.328039 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:44.328049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:44.328071 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:44.328127 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:44.332280 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:44.333012 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:44.333021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:44.333032 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:44.333254 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:44.333606 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:44.333617 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:44.333621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:44.334385 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:44.334722 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:44.335101 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 16:54:44.335607 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":395,"threads_started":1}
I20260619 16:54:44.337739 4602 consensus_queue.cc:260] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f13d40f8e43e09533f077b3d5df2c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44227 } }
I20260619 16:54:44.338013 4602 raft_consensus.cc:388] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:44.338176 4602 raft_consensus.cc:482] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:44.338375 4602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:44.341009 4602 raft_consensus.cc:504] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f13d40f8e43e09533f077b3d5df2c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44227 } }
I20260619 16:54:44.341202 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:44.341609 4602 leader_election.cc:302] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a44f13d40f8e43e09533f077b3d5df2c; no voters:
I20260619 16:54:44.342250 4602 leader_election.cc:288] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:44.342372 4607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:44.343827 4607 raft_consensus.cc:686] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [term 1 LEADER]: Becoming Leader. State: Replica: a44f13d40f8e43e09533f077b3d5df2c, State: Running, Role: LEADER
I20260619 16:54:44.344340 4607 consensus_queue.cc:237] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f13d40f8e43e09533f077b3d5df2c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44227 } }
I20260619 16:54:44.344539 4602 sys_catalog.cc:564] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:44.347626 4609 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a44f13d40f8e43e09533f077b3d5df2c. Latest consensus state: current_term: 1 leader_uuid: "a44f13d40f8e43e09533f077b3d5df2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f13d40f8e43e09533f077b3d5df2c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44227 } } }
I20260619 16:54:44.348047 4609 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:44.348031 4608 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a44f13d40f8e43e09533f077b3d5df2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44f13d40f8e43e09533f077b3d5df2c" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 44227 } } }
I20260619 16:54:44.348323 4608 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:44.352392 4618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:44.358577 4618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:44.368963 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:44.349193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36543 (local address 127.14.153.62:44227)
0619 16:54:44.349435 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:44.349445 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:44.350109 (+ 664us) server_negotiation.cc:407] Connection header received
0619 16:54:44.350428 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:44.350436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:44.350546 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:44.350668 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:44.353188 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:44.354347 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:44.356320 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:44.357068 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:44.357187 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:44.360711 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:44.360719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:44.360727 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:44.360767 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:44.364665 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:44.365564 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:44.365569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:44.365572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:44.365741 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:44.366199 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:44.366203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:44.366205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:44.366628 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:44.366864 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:44.368097 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0619 16:54:44.368404 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:54:44.371706 4618 catalog_manager.cc:1347] Generated new cluster ID: 17723dc7b08844378d30d3c476bd1aa3
I20260619 16:54:44.371817 4618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:44.389017 4618 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:44.390432 4618 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:44.399058 4618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a44f13d40f8e43e09533f077b3d5df2c: Generated new TSK 0
I20260619 16:54:44.400307 4618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:44.414067 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4535
I20260619 16:54:44.415299 4597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:44.505569 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 4535
2026-06-19T16:54:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:44 dist-test-slave-qb36 krb5kdc[4634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:44 dist-test-slave-qb36 krb5kdc[4634](info): set up 2 sockets
Jun 19 16:54:44 dist-test-slave-qb36 krb5kdc[4634](info): commencing operation
krb5kdc: starting...
W20260619 16:54:45.615589 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:45 dist-test-slave-qb36 krb5kdc[4634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:45.878890 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:40499
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:40499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:46.216300 4647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:46.216625 4647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:46.216725 4647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:46.224161 4647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:46.224282 4647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:46.224328 4647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:46.224390 4647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:46.224434 4647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:46.234493 4647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:40499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:40499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:46.235833 4647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:46.237119 4647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:46.249109 4653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:46.249065 4655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:46.249487 4652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:46.250890 4647 server_base.cc:1034] running on GCE node
I20260619 16:54:46.251811 4647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:46.253363 4647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:46.254602 4647 hybrid_clock.cc:648] HybridClock initialized: now 1781888086254560 us; error 64 us; skew 500 ppm
I20260619 16:54:46.254971 4647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:46 dist-test-slave-qb36 krb5kdc[4634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:46.258133 4647 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:46.259639 4647 webserver.cc:460] Webserver started at http://127.14.153.62:43251/ using document root <none> and password file <none>
I20260619 16:54:46.260349 4647 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:46.260495 4647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:46.260805 4647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:46.263942 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "b127c78a23e140d0b55c76845cfac8e2"
format_stamp: "Formatted at 2026-06-19 16:54:46 on dist-test-slave-qb36"
server_key: "16a51c39b44da3e19fd19e1fdc6a2b74"
server_key_iv: "2bd8dc806dda2ecde68c9bd002b638a6"
server_key_version: "encryptionkey@0"
I20260619 16:54:46.264714 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "b127c78a23e140d0b55c76845cfac8e2"
format_stamp: "Formatted at 2026-06-19 16:54:46 on dist-test-slave-qb36"
server_key: "16a51c39b44da3e19fd19e1fdc6a2b74"
server_key_iv: "2bd8dc806dda2ecde68c9bd002b638a6"
server_key_version: "encryptionkey@0"
I20260619 16:54:46.269903 4647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 16:54:46.273471 4662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:46.274873 4647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 16:54:46.275116 4647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "b127c78a23e140d0b55c76845cfac8e2"
format_stamp: "Formatted at 2026-06-19 16:54:46 on dist-test-slave-qb36"
server_key: "16a51c39b44da3e19fd19e1fdc6a2b74"
server_key_iv: "2bd8dc806dda2ecde68c9bd002b638a6"
server_key_version: "encryptionkey@0"
I20260619 16:54:46.275345 4647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:46.291148 4647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:46.295615 4647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:46.295941 4647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:46.315011 4647 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:40499
I20260619 16:54:46.315063 4713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:40499 every 8 connection(s)
I20260619 16:54:46.316545 4647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:46.320389 4714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:46.325016 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 4647
I20260619 16:54:46.325237 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:46.325665 14948 external_mini_cluster.cc:1428] Setting key 3c8f36139e6789cbb5fbb435f640015e
I20260619 16:54:46.333153 4714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2: Bootstrap starting.
I20260619 16:54:46.336946 4714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:46.338318 4714 log.cc:824] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:46.341537 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:46.327635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42239 (local address 127.14.153.62:40499)
0619 16:54:46.328529 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:46.328558 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:46.328616 (+ 58us) server_negotiation.cc:407] Connection header received
0619 16:54:46.331278 (+ 2662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:46.331309 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:46.331876 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:46.332342 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:46.332877 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:46.334865 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:46.336294 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:46.337426 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:46.337636 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:46.338770 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:46.338780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:46.338799 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:46.339516 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:46.339595 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:46.339620 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:46.339758 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:46.339979 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 16:54:46.340250 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":351,"threads_started":1}
I20260619 16:54:46.341833 4714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2: No bootstrap required, opened a new log
I20260619 16:54:46.346855 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:46.350258 4714 raft_consensus.cc:348] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b127c78a23e140d0b55c76845cfac8e2" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40499 } }
I20260619 16:54:46.350642 4714 raft_consensus.cc:374] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:46.350741 4714 raft_consensus.cc:729] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b127c78a23e140d0b55c76845cfac8e2, State: Initialized, Role: FOLLOWER
I20260619 16:54:46.351596 4714 consensus_queue.cc:260] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b127c78a23e140d0b55c76845cfac8e2" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40499 } }
I20260619 16:54:46.351897 4714 raft_consensus.cc:388] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:46.352057 4714 raft_consensus.cc:482] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:46.352237 4714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:46.355428 4714 raft_consensus.cc:504] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b127c78a23e140d0b55c76845cfac8e2" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40499 } }
I20260619 16:54:46.356045 4714 leader_election.cc:302] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b127c78a23e140d0b55c76845cfac8e2; no voters:
I20260619 16:54:46.356745 4714 leader_election.cc:288] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:46.356856 4724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:46.357847 4724 raft_consensus.cc:686] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [term 1 LEADER]: Becoming Leader. State: Replica: b127c78a23e140d0b55c76845cfac8e2, State: Running, Role: LEADER
I20260619 16:54:46.358469 4724 consensus_queue.cc:237] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b127c78a23e140d0b55c76845cfac8e2" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40499 } }
I20260619 16:54:46.358975 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:46.351258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53387 (local address 127.14.153.62:40499)
0619 16:54:46.351623 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:46.351632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:46.352228 (+ 596us) server_negotiation.cc:407] Connection header received
0619 16:54:46.352304 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:46.352308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:46.352403 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:46.352487 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:46.353128 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:46.353884 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:46.355769 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:46.356313 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:46.356411 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:46.357743 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:46.357754 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:46.357762 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:46.357787 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:46.357848 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:46.357872 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:46.358484 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:46.358535 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:54:46.358641 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260619 16:54:46.359193 4714 sys_catalog.cc:564] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:46.362098 4725 sys_catalog.cc:455] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b127c78a23e140d0b55c76845cfac8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b127c78a23e140d0b55c76845cfac8e2" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40499 } } }
I20260619 16:54:46.362075 4726 sys_catalog.cc:455] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b127c78a23e140d0b55c76845cfac8e2. Latest consensus state: current_term: 1 leader_uuid: "b127c78a23e140d0b55c76845cfac8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b127c78a23e140d0b55c76845cfac8e2" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 40499 } } }
I20260619 16:54:46.362497 4725 sys_catalog.cc:458] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:46.362572 4726 sys_catalog.cc:458] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:46.364737 4730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:46.369050 4730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:46.378312 4730 catalog_manager.cc:1347] Generated new cluster ID: a1f021f571e24d6490fbfbe8575bd771
I20260619 16:54:46.378435 4730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:46.400650 4730 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:46.402319 4730 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:46.408723 4730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b127c78a23e140d0b55c76845cfac8e2: Generated new TSK 0
I20260619 16:54:46.409615 4730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:46.419693 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4647
W20260619 16:54:46.420964 4709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42239
W20260619 16:54:46.421734 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4647: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:46.421918 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 4647
2026-06-19T16:54:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:46 dist-test-slave-qb36 krb5kdc[4745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:46 dist-test-slave-qb36 krb5kdc[4745](info): set up 2 sockets
Jun 19 16:54:46 dist-test-slave-qb36 krb5kdc[4745](info): commencing operation
krb5kdc: starting...
W20260619 16:54:47.539762 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:47 dist-test-slave-qb36 krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:47.806900 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34165
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:48.142939 4758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:48.143350 4758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:48.143518 4758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:48.151096 4758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:48.151238 4758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:48.151325 4758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:48.151414 4758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:48.151497 4758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:48.161878 4758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:48.163285 4758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:48.164624 4758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:48.176724 4766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:48.176848 4763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:48.176829 4764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:48.177692 4758 server_base.cc:1034] running on GCE node
I20260619 16:54:48.178493 4758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:48.180051 4758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:48.181301 4758 hybrid_clock.cc:648] HybridClock initialized: now 1781888088181266 us; error 52 us; skew 500 ppm
I20260619 16:54:48.181670 4758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:48 dist-test-slave-qb36 krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888088, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:48.188138 4758 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:48.189817 4758 webserver.cc:460] Webserver started at http://127.14.153.62:35193/ using document root <none> and password file <none>
I20260619 16:54:48.190584 4758 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:48.190732 4758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:48.191110 4758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:48.194151 4758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "8afcd2cf624740be8c90ba58a4251d34"
format_stamp: "Formatted at 2026-06-19 16:54:48 on dist-test-slave-qb36"
server_key: "6bebcf0c0e6cdfc78b8c26be6c2f1745"
server_key_iv: "dafd60add5bd9e8eb2c72ca357a249ba"
server_key_version: "encryptionkey@0"
I20260619 16:54:48.194916 4758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "8afcd2cf624740be8c90ba58a4251d34"
format_stamp: "Formatted at 2026-06-19 16:54:48 on dist-test-slave-qb36"
server_key: "6bebcf0c0e6cdfc78b8c26be6c2f1745"
server_key_iv: "dafd60add5bd9e8eb2c72ca357a249ba"
server_key_version: "encryptionkey@0"
I20260619 16:54:48.200208 4758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:54:48.203823 4773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:48.205210 4758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:48.205411 4758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "8afcd2cf624740be8c90ba58a4251d34"
format_stamp: "Formatted at 2026-06-19 16:54:48 on dist-test-slave-qb36"
server_key: "6bebcf0c0e6cdfc78b8c26be6c2f1745"
server_key_iv: "dafd60add5bd9e8eb2c72ca357a249ba"
server_key_version: "encryptionkey@0"
I20260619 16:54:48.205628 4758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:48.211726 4758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:48.216253 4758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:48.216562 4758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:48.235553 4758 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:34165
I20260619 16:54:48.235576 4824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:34165 every 8 connection(s)
I20260619 16:54:48.236985 4758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:48.240682 4825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:48.242164 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 4758
I20260619 16:54:48.242359 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:48.242878 14948 external_mini_cluster.cc:1428] Setting key 41c1e5262446f5eda1a60c9446053d6f
I20260619 16:54:48.250563 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:48.244726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34511 (local address 127.14.153.62:34165)
0619 16:54:48.245536 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:48.245548 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:48.245589 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:54:48.247237 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:48.247260 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:48.247608 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:48.247942 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:48.249077 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:48.249083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:48.249095 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:48.249109 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:48.249164 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:48.249187 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:48.249342 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:48.249479 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 16:54:48.249682 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":370,"threads_started":1}
I20260619 16:54:48.254215 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:48.254755 4825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34: Bootstrap starting.
I20260619 16:54:48.258107 4825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:48.259521 4825 log.cc:824] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:48.261077 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:48.258581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38709 (local address 127.14.153.62:34165)
0619 16:54:48.258765 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:48.258771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:48.259154 (+ 383us) server_negotiation.cc:407] Connection header received
0619 16:54:48.259377 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:48.259382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:48.259437 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:48.259546 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:48.260461 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:48.260469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:48.260474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:54:48.260491 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:48.260538 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:54:48.260553 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:48.260668 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:48.260771 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 16:54:48.260826 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:54:48.262630 4825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34: No bootstrap required, opened a new log
I20260619 16:54:48.270434 4825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcd2cf624740be8c90ba58a4251d34" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34165 } }
I20260619 16:54:48.270954 4825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:54:48.271097 4825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afcd2cf624740be8c90ba58a4251d34, State: Initialized, Role: FOLLOWER
I20260619 16:54:48.271651 4825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcd2cf624740be8c90ba58a4251d34" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34165 } }
I20260619 16:54:48.271854 4825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:54:48.271943 4825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:54:48.272060 4825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:48.274138 4825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcd2cf624740be8c90ba58a4251d34" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34165 } }
I20260619 16:54:48.274569 4825 leader_election.cc:302] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8afcd2cf624740be8c90ba58a4251d34; no voters:
I20260619 16:54:48.275184 4825 leader_election.cc:288] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:54:48.275321 4835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:48.276026 4835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [term 1 LEADER]: Becoming Leader. State: Replica: 8afcd2cf624740be8c90ba58a4251d34, State: Running, Role: LEADER
I20260619 16:54:48.276618 4835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcd2cf624740be8c90ba58a4251d34" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34165 } }
I20260619 16:54:48.277326 4825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:48.279646 4836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8afcd2cf624740be8c90ba58a4251d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcd2cf624740be8c90ba58a4251d34" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34165 } } }
I20260619 16:54:48.280459 4836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:48.280155 4837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8afcd2cf624740be8c90ba58a4251d34. Latest consensus state: current_term: 1 leader_uuid: "8afcd2cf624740be8c90ba58a4251d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afcd2cf624740be8c90ba58a4251d34" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34165 } } }
I20260619 16:54:48.282470 4837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:48.283592 4843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:48.288882 4843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:48.297593 4843 catalog_manager.cc:1347] Generated new cluster ID: 15882d9e51cd4f7081051aaa2ce08d2c
I20260619 16:54:48.297729 4843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:48.317881 4843 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:48.318410 4843 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:48.328507 4843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8afcd2cf624740be8c90ba58a4251d34: Generated new TSK 0
I20260619 16:54:48.329418 4843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:48.338864 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4758
W20260619 16:54:48.340111 4820 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34511
W20260619 16:54:48.340880 14948 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4758: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 16:54:48.341018 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 4758
2026-06-19T16:54:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1917 ms)
[----------] 18 tests from AuthTokenIssuingTest (35285 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:48 dist-test-slave-qb36 krb5kdc[4856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:48 dist-test-slave-qb36 krb5kdc[4856](info): set up 2 sockets
Jun 19 16:54:48 dist-test-slave-qb36 krb5kdc[4856](info): commencing operation
krb5kdc: starting...
W20260619 16:54:49.443861 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:49 dist-test-slave-qb36 krb5kdc[4856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:49.712224 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:38487
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:38487,127.14.153.61:40777,127.14.153.60:40021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:50.053869 4869 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:54:50.054306 4869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:50.054440 4869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:50.054580 4869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:50.062417 4869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:50.062567 4869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:50.062685 4869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:50.062778 4869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:50.062868 4869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:50.073123 4869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:38487,127.14.153.61:40777,127.14.153.60:40021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:38487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:50.074559 4869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:50.075980 4869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:50.088541 4877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:50.088640 4875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:50.088565 4874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:50.092727 4869 server_base.cc:1034] running on GCE node
I20260619 16:54:50.093618 4869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:50.095225 4869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:50.096490 4869 hybrid_clock.cc:648] HybridClock initialized: now 1781888090096411 us; error 62 us; skew 500 ppm
I20260619 16:54:50.096903 4869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:50 dist-test-slave-qb36 krb5kdc[4856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:50.100468 4869 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:50.102072 4869 webserver.cc:460] Webserver started at http://127.14.153.62:44153/ using document root <none> and password file <none>
I20260619 16:54:50.102886 4869 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:50.103020 4869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:50.103408 4869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:50.106494 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "5c305344369044379ea5d7151c43e4d6"
format_stamp: "Formatted at 2026-06-19 16:54:50 on dist-test-slave-qb36"
server_key: "70f4d25d46818986a088bb539024f8c4"
server_key_iv: "6a03a725f0fb1c3e408a6b7af05a990c"
server_key_version: "encryptionkey@0"
I20260619 16:54:50.107465 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "5c305344369044379ea5d7151c43e4d6"
format_stamp: "Formatted at 2026-06-19 16:54:50 on dist-test-slave-qb36"
server_key: "70f4d25d46818986a088bb539024f8c4"
server_key_iv: "6a03a725f0fb1c3e408a6b7af05a990c"
server_key_version: "encryptionkey@0"
I20260619 16:54:50.112819 4869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:54:50.116662 4884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:50.118227 4869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:50.118435 4869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "5c305344369044379ea5d7151c43e4d6"
format_stamp: "Formatted at 2026-06-19 16:54:50 on dist-test-slave-qb36"
server_key: "70f4d25d46818986a088bb539024f8c4"
server_key_iv: "6a03a725f0fb1c3e408a6b7af05a990c"
server_key_version: "encryptionkey@0"
I20260619 16:54:50.118695 4869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:50.150060 4869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:50.154856 4869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:50.155184 4869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:50.174752 4869 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:38487
I20260619 16:54:50.174793 4935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:38487 every 8 connection(s)
I20260619 16:54:50.176319 4869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:50.179095 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 4869
I20260619 16:54:50.179319 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:50.179980 14948 external_mini_cluster.cc:1428] Setting key 5adef8776caba3ac8aa29179ba0ed2ee
I20260619 16:54:50.180509 4936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:50.187367 4936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:50 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.61@KRBTEST.COM" created.
I20260619 16:54:50.212532 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.190583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:38487 (local address 127.0.0.1:56438)
0619 16:54:50.191437 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.191484 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:50.193631 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:50.194628 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:50.194634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:50.195286 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:50.195648 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:50.195666 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.198048 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.198052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:50.199216 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:50.199229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.200645 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.200650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:50.200801 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:50.201515 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:50.201537 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:50.203789 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:50.208144 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:50.208155 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:50.208166 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:50.208566 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:50.209069 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:50.209073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:50.209075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:50.209226 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:50.210257 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:50.210263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:50.210447 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 16:54:50.210658 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 16:54:50.211196 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":393,"threads_started":1}
I20260619 16:54:50.212559 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.191942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.14.153.62:38487)
0619 16:54:50.192691 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:50.192701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:50.192760 (+ 59us) server_negotiation.cc:407] Connection header received
0619 16:54:50.194039 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:50.194049 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:50.194372 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:50.194597 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:50.195881 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.197879 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.199365 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.200416 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.200648 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:50.204029 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:50.204038 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:50.204051 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:50.204081 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:50.207974 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:50.208732 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:50.208744 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:50.208750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:50.208957 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:50.209352 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:50.209359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:50.209365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:50.210122 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:50.210351 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:50.210860 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 16:54:50.211417 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":497,"thread_start_us":325,"threads_started":1}
I20260619 16:54:50.215888 4936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:54:50.217182 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.216691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41090)
0619 16:54:50.216883 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.217023 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 16:54:50.217525 4886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
I20260619 16:54:50.219380 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.218821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41098)
0619 16:54:50.219017 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.219200 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 16:54:50.219605 4936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:40777: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab.
Entry for principal kudu/127.14.153.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab.
I20260619 16:54:50.238741 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.61:40777
--webserver_interface=127.14.153.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab
--principal=kudu/127.14.153.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:38487,127.14.153.61:40777,127.14.153.60:40021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:54:50.268987 4936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } attempt: 1
I20260619 16:54:50.270273 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.269740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41112)
0619 16:54:50.269968 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.270097 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 16:54:50.271400 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.270976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41124)
0619 16:54:50.271152 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.271240 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 16:54:50.271692 4936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:40777: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
I20260619 16:54:50.339967 4936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } attempt: 2
I20260619 16:54:50.341276 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.340760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41138)
0619 16:54:50.341020 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.341129 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 16:54:50.342424 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.342000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41154)
0619 16:54:50.342199 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.342286 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 16:54:50.342614 4936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:40777: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
I20260619 16:54:50.433873 4936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } attempt: 3
I20260619 16:54:50.435328 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.434738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41164)
0619 16:54:50.434976 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.435186 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 16:54:50.436480 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.436080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41178)
0619 16:54:50.436265 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.436336 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 16:54:50.436648 4936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:40777: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
W20260619 16:54:50.573599 4943 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:54:50.574029 4943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:50.574163 4943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:50.574311 4943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:54:50.579880 4936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } attempt: 4
I20260619 16:54:50.581218 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.580627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41192)
0619 16:54:50.580973 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.581081 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260619 16:54:50.582330 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.581898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41204)
0619 16:54:50.582068 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.582149 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 16:54:50.582458 4943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:50.582532 4936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:40777: Network error: Client connection negotiation failed: client connection to 127.14.153.61:40777: connect: Connection refused (error 111)
W20260619 16:54:50.582574 4943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:50.582690 4943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:50.582787 4943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:50.582875 4943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:50.593253 4943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:38487,127.14.153.61:40777,127.14.153.60:40021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab
--principal=kudu/127.14.153.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.61:40777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.153.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:50.594669 4943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:50.596084 4943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:50.608099 4949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:50.608057 4948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:50.608132 4951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:50.608994 4943 server_base.cc:1034] running on GCE node
I20260619 16:54:50.609827 4943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:50.611510 4943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:50.612761 4943 hybrid_clock.cc:648] HybridClock initialized: now 1781888090612675 us; error 57 us; skew 500 ppm
I20260619 16:54:50.613108 4943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:50 dist-test-slave-qb36 krb5kdc[4856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:50.616380 4943 init.cc:377] Logged in from keytab as kudu/127.14.153.61@KRBTEST.COM (short username kudu)
I20260619 16:54:50.617956 4943 webserver.cc:460] Webserver started at http://127.14.153.61:33359/ using document root <none> and password file <none>
I20260619 16:54:50.618646 4943 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:50.618738 4943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:50.619010 4943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:50.622176 4943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/data/instance:
uuid: "f1c40cb3595f45d391d76efda824dcce"
format_stamp: "Formatted at 2026-06-19 16:54:50 on dist-test-slave-qb36"
server_key: "53274155c020bf58a92745d8dffd8b49"
server_key_iv: "a8380210998df934aa650d3f54420094"
server_key_version: "encryptionkey@0"
I20260619 16:54:50.622987 4943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/wal/instance:
uuid: "f1c40cb3595f45d391d76efda824dcce"
format_stamp: "Formatted at 2026-06-19 16:54:50 on dist-test-slave-qb36"
server_key: "53274155c020bf58a92745d8dffd8b49"
server_key_iv: "a8380210998df934aa650d3f54420094"
server_key_version: "encryptionkey@0"
I20260619 16:54:50.628403 4943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 16:54:50.631902 4958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:50.633333 4943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:50.633507 4943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/wal
uuid: "f1c40cb3595f45d391d76efda824dcce"
format_stamp: "Formatted at 2026-06-19 16:54:50 on dist-test-slave-qb36"
server_key: "53274155c020bf58a92745d8dffd8b49"
server_key_iv: "a8380210998df934aa650d3f54420094"
server_key_version: "encryptionkey@0"
I20260619 16:54:50.633733 4943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:50.667781 4943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:50.672281 4943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:50.672565 4943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:50.692021 4943 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.61:40777
I20260619 16:54:50.692028 5009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.61:40777 every 8 connection(s)
I20260619 16:54:50.693472 4943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
I20260619 16:54:50.694676 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 4943
I20260619 16:54:50.694873 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-1/wal/instance
I20260619 16:54:50.695459 14948 external_mini_cluster.cc:1428] Setting key 790d6b7fea0a9572830d6ff2f5d7a163
I20260619 16:54:50.698343 5010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:50.707798 5010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.153.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.60@KRBTEST.COM" created.
Jun 19 16:54:50 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:50.734076 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.710414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.14.153.62:38487)
0619 16:54:50.710680 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:50.710689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:50.711963 (+ 1274us) server_negotiation.cc:407] Connection header received
0619 16:54:50.714899 (+ 2936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:50.714903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:50.714976 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:50.715144 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:50.717264 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.717940 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.720568 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.720986 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.721083 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:50.728032 (+ 6949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:50.728037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:50.728041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:50.728076 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:50.730786 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:50.731675 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:50.731679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:50.731682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:50.731806 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:50.732275 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:50.732285 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:50.732287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:50.732565 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:50.732752 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:50.733489 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 16:54:50.733742 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 16:54:50.735129 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.710886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:38487 (local address 127.0.0.1:56452)
0619 16:54:50.711763 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.711818 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:50.714576 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:50.715383 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:50.715397 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:50.716232 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:50.717073 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:50.717104 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.718114 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.718123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:50.720390 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:50.720411 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.721225 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.721232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:50.721553 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:50.722363 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:50.722399 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:50.727776 (+ 5377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:50.730953 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:50.730963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:50.730982 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:50.731521 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:50.731930 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:50.731935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:50.731939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:50.732142 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:50.732712 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:50.732722 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:50.733107 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 16:54:50.733393 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 16:54:50.733947 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":360,"threads_started":1}
I20260619 16:54:50.736945 5010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:50 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
I20260619 16:54:50.753214 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.737837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41220)
0619 16:54:50.737991 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.738013 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:50.738116 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:50.739313 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:50.739321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:50.739853 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:50.740070 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:50.740084 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.741427 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.741440 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:50.743050 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:50.743069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.743884 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.743893 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:50.744123 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:50.744757 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:50.744785 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:50.747425 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:50.750649 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:50.750662 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:50.750666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:50.751080 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:50.751460 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:50.751465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:50.751468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:50.751591 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:50.752199 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:50.752204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:50.752347 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 16:54:50.752490 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 16:54:50.752800 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 16:54:50.753328 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.738102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.14.153.61:40777)
0619 16:54:50.738827 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:50.738833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:50.738858 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:54:50.738900 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:50.738906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:50.739142 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:50.739261 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:50.740247 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.741283 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.743187 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.743725 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.743821 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:50.747631 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:50.747643 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:50.747659 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:50.747691 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:50.750477 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:50.751201 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:50.751216 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:50.751218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:50.751334 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:50.751714 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:50.751720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:50.751728 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:50.752071 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:50.752300 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:50.752636 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 16:54:50.752919 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":556,"thread_start_us":410,"threads_started":1}
I20260619 16:54:50.756069 5010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:54:50.757467 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.756878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60792)
0619 16:54:50.757164 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.757312 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab.
Entry for principal kudu/127.14.153.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab.
W20260619 16:54:50.757879 4961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
I20260619 16:54:50.758813 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.60:40021
--webserver_interface=127.14.153.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab
--principal=kudu/127.14.153.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:38487,127.14.153.61:40777,127.14.153.60:40021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:54:50.759493 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.759143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60800)
0619 16:54:50.759316 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.759392 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:54:50.759732 5010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:40021: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
I20260619 16:54:50.809016 5010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } attempt: 1
I20260619 16:54:50.810235 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.809761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60812)
0619 16:54:50.809961 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.810066 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 16:54:50.811268 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.810883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60814)
0619 16:54:50.811072 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.811160 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 16:54:50.811475 5010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:40021: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
I20260619 16:54:50.879683 5010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } attempt: 2
I20260619 16:54:50.880776 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.880316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60828)
0619 16:54:50.880518 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.880617 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 16:54:50.881771 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.881409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60830)
0619 16:54:50.881544 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.881623 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:54:50.881765 4936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } attempt: 5
W20260619 16:54:50.882193 5010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:40021: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Jun 19 16:54:50 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
I20260619 16:54:50.895383 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.882371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41230)
0619 16:54:50.882559 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.882575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:50.882677 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:50.883149 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:50.883156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:50.883676 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:50.883903 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:50.883914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.884889 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.884894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:50.885764 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:50.885775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.886526 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.886531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:50.886751 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:50.887421 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:50.887438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:50.889673 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:50.892878 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:50.892884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:50.892888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:50.893260 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:50.893647 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:50.893660 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:50.893664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:50.893811 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:50.894349 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:50.894354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:50.894517 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 16:54:50.894671 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 16:54:50.894986 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:54:50.895385 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.882377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.14.153.61:40777)
0619 16:54:50.882834 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:50.882839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:50.882852 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:54:50.882896 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:50.882900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:50.882969 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:50.883106 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:50.884048 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.884758 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.885909 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:50.886378 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:50.886513 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:50.889875 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:50.889881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:50.889884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:50.889916 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:50.892726 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:50.893388 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:50.893396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:50.893399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:50.893536 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:50.893921 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:50.893927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:50.893930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:50.894230 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:50.894385 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:50.894771 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 16:54:50.894993 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375}
I20260619 16:54:50.896770 4936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:54:50.897964 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.897503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60832)
0619 16:54:50.897698 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.897792 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 16:54:50.898955 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.898575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60840)
0619 16:54:50.898734 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.898833 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 16:54:50.899183 4936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:40021: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
I20260619 16:54:50.950426 4936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } attempt: 1
I20260619 16:54:50.951715 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.951158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60852)
0619 16:54:50.951378 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.951490 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 16:54:50.952894 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.952463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60864)
0619 16:54:50.952677 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.952767 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 16:54:50.953142 4936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:40021: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
I20260619 16:54:50.973435 5010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } attempt: 3
I20260619 16:54:50.974628 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.974132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60878)
0619 16:54:50.974367 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.974460 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 16:54:50.975797 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:50.975347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60882)
0619 16:54:50.975616 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:50.975692 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260619 16:54:50.975997 5010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:40021: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
I20260619 16:54:51.021371 4936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } attempt: 2
I20260619 16:54:51.022754 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.022131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60888)
0619 16:54:51.022399 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.022527 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 16:54:51.024022 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.023566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60898)
0619 16:54:51.023768 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.023878 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 16:54:51.024300 4936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:40021: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
W20260619 16:54:51.099699 5017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:54:51.100033 5017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:51.100096 5017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:51.100172 5017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:51.107618 5017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:51.107717 5017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:51.107775 5017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:51.107825 5017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:51.107882 5017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:51.117898 5017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:38487,127.14.153.61:40777,127.14.153.60:40021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab
--principal=kudu/127.14.153.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.60:40021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.153.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:51.119249 5017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:51.119208 5010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } attempt: 4
I20260619 16:54:51.120457 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.119864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60902)
0619 16:54:51.120147 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.120259 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 16:54:51.120916 5017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:54:51.121610 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.121213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60912)
0619 16:54:51.121380 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.121475 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 16:54:51.121867 5010 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:40021: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
I20260619 16:54:51.130519 4936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } attempt: 3
I20260619 16:54:51.131806 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.131283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60928)
0619 16:54:51.131469 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.131575 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 16:54:51.133133 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.132542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60942)
0619 16:54:51.132871 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.132966 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260619 16:54:51.133422 4936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:40021: Network error: Client connection negotiation failed: client connection to 127.14.153.60:40021: connect: Connection refused (error 111)
W20260619 16:54:51.133893 5022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:51.133854 5025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:51.134251 5023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:51.135350 5017 server_base.cc:1034] running on GCE node
I20260619 16:54:51.136183 5017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:51.137674 5017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:51.138906 5017 hybrid_clock.cc:648] HybridClock initialized: now 1781888091138834 us; error 42 us; skew 500 ppm
I20260619 16:54:51.139283 5017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:51 dist-test-slave-qb36 krb5kdc[4856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888091, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:51.142454 5017 init.cc:377] Logged in from keytab as kudu/127.14.153.60@KRBTEST.COM (short username kudu)
I20260619 16:54:51.144223 5017 webserver.cc:460] Webserver started at http://127.14.153.60:45035/ using document root <none> and password file <none>
I20260619 16:54:51.144949 5017 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:51.145066 5017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:51.145393 5017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:51.148459 5017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/data/instance:
uuid: "5181fca92fc74b2d9afcd94ab34c4cdd"
format_stamp: "Formatted at 2026-06-19 16:54:51 on dist-test-slave-qb36"
server_key: "48ab52e94fce4d62c264546b0d271ef5"
server_key_iv: "d8051ac79de53792f93e19a83ad642df"
server_key_version: "encryptionkey@0"
I20260619 16:54:51.149278 5017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/wal/instance:
uuid: "5181fca92fc74b2d9afcd94ab34c4cdd"
format_stamp: "Formatted at 2026-06-19 16:54:51 on dist-test-slave-qb36"
server_key: "48ab52e94fce4d62c264546b0d271ef5"
server_key_iv: "d8051ac79de53792f93e19a83ad642df"
server_key_version: "encryptionkey@0"
I20260619 16:54:51.154593 5017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 16:54:51.158203 5032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:51.159639 5017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:54:51.159855 5017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/wal
uuid: "5181fca92fc74b2d9afcd94ab34c4cdd"
format_stamp: "Formatted at 2026-06-19 16:54:51 on dist-test-slave-qb36"
server_key: "48ab52e94fce4d62c264546b0d271ef5"
server_key_iv: "d8051ac79de53792f93e19a83ad642df"
server_key_version: "encryptionkey@0"
I20260619 16:54:51.160074 5017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:51.201421 5017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:51.205883 5017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:51.206177 5017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:51.225224 5017 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.60:40021
I20260619 16:54:51.225243 5083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.60:40021 every 8 connection(s)
I20260619 16:54:51.226661 5017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
I20260619 16:54:51.230605 5084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:51.234994 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 5017
I20260619 16:54:51.235234 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887734944185-14948-0/minicluster-data/master-2/wal/instance
I20260619 16:54:51.235693 14948 external_mini_cluster.cc:1428] Setting key 628178c365e46748e84e7e41270d34df
I20260619 16:54:51.237905 5084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:51 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888091, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
Jun 19 16:54:51 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
I20260619 16:54:51.257649 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.240812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.14.153.62:38487)
0619 16:54:51.241443 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.241450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.242075 (+ 625us) server_negotiation.cc:407] Connection header received
0619 16:54:51.242150 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.242157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.242247 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.242365 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.243671 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.244457 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.246058 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.246479 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.246577 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.250352 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.250360 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.250365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.250408 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.253985 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.254840 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.254845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.254849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.255011 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.255618 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.255621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.255623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.255936 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.256094 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.256985 (+ 891us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.257246 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":316,"threads_started":1}
I20260619 16:54:51.258726 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.240826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:38487 (local address 127.0.0.1:56454)
0619 16:54:51.241725 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.241764 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:51.241945 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:51.242439 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:51.242444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:51.243021 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:51.243487 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.243503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.244611 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.244617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.245901 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.245913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.246717 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.246725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.246877 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.247590 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:51.247613 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:51.250085 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:51.254141 (+ 4056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.254151 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.254161 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.254712 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.255203 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.255208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.255212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.255477 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.256106 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:51.256112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:51.256611 (+ 499us) client_negotiation.cc:769] Sending connection context
0619 16:54:51.256854 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 16:54:51.257376 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":568,"threads_started":1}
I20260619 16:54:51.258773 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.237735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60956 (local address 127.14.153.60:40021)
0619 16:54:51.238774 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.238786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.238829 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:54:51.240882 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.240912 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.241528 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.241925 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.243487 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.244774 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.246042 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.246608 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.246803 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.251431 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.251445 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.251465 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.251503 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.255095 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.255936 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.255946 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.255949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.256103 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.256524 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.256528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.256530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.256938 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.257246 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.257586 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.257858 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":502,"threads_started":1}
I20260619 16:54:51.260520 5084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:51 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888091, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
Jun 19 16:54:51 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:51.275398 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.261189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:40777 (local address 127.0.0.1:41238)
0619 16:54:51.261334 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.261350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:51.261468 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:51.261834 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:51.261838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:51.262314 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:51.262475 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.262484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.263972 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.263976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.265357 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.265373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.266323 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.266327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.266463 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.267164 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:51.267186 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:51.269448 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:51.272935 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.272941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.272945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.273310 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.273699 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.273704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.273708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.273855 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.274375 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:51.274380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:51.274546 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 16:54:51.274695 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 16:54:51.274982 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 16:54:51.275424 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.261323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.14.153.61:40777)
0619 16:54:51.261512 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.261520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.261540 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:54:51.261622 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.261627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.261704 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.261801 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.262633 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.263795 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.265520 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.266176 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.266311 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.269645 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.269653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.269658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.269702 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.272787 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.273430 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.273436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.273440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.273577 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.273965 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.273971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.273975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.274252 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.274409 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.274797 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.275002 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:54:51.276445 5084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:54:51.278610 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.262308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.14.153.62:38487)
0619 16:54:51.262482 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.262487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.262504 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:54:51.262662 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.262666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.262739 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.262812 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.264306 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.265437 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.267189 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.267835 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.267960 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.272491 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.272496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.272499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.272530 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.275221 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.276056 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.276062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.276065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.276230 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.276713 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.276718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.276721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.277019 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.277300 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.278204 (+ 904us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.278386 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 16:54:51 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888091, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
Jun 19 16:54:51 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
I20260619 16:54:51.290961 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.277549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.14.153.60:40021)
0619 16:54:51.277716 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.277722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.277737 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:54:51.277785 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.277793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.277870 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.277965 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.278753 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.279659 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.280967 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.281380 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.281512 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.284898 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.284903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.284906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.284940 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.288352 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.289124 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.289128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.289130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.289251 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.289629 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.289637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.289643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.289941 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.290096 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.290490 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.290714 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":44}
I20260619 16:54:51.291296 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.277304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60964)
0619 16:54:51.277472 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.277491 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:51.277590 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:51.278066 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:51.278070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:51.278453 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:51.278620 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.278628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.279784 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.279788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.280832 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.280843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.281519 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.281524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.281727 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.282289 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:51.282305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:51.284685 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:51.288499 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.288503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.288505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.288990 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.289371 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.289379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.289384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.289522 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.290058 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:51.290062 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:51.290245 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 16:54:51.290404 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 16:54:51.290915 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":133}
I20260619 16:54:51.295970 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.280235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.14.153.61:40777)
0619 16:54:51.280428 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.280434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.280451 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:54:51.280662 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.280668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.280741 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.280837 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.281981 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.282689 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.284483 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.285077 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.285174 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.290309 (+ 5135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.290316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.290320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.290363 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.293322 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.294160 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.294164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.294166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.294317 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.294704 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.294708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.294712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.294951 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.295239 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.295525 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.295706 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:54:51.297415 5084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd: Bootstrap starting.
I20260619 16:54:51.301082 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:51.301343 5084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:51.302412 5084 log.cc:824] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:51.305352 5084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd: No bootstrap required, opened a new log
I20260619 16:54:51.303913 4911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5c305344369044379ea5d7151c43e4d6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56456
I20260619 16:54:51.310643 4936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } attempt: 4
I20260619 16:54:51.313311 5084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } }
I20260619 16:54:51.313750 5084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5181fca92fc74b2d9afcd94ab34c4cdd, State: Initialized, Role: FOLLOWER
I20260619 16:54:51.314292 5084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } }
I20260619 16:54:51.315387 5091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.315855 5091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:51.316707 5084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 16:54:51 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
I20260619 16:54:51.327579 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.311631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.14.153.60:40021)
0619 16:54:51.311853 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.311862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.311879 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:54:51.311941 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.311945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.312022 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.312128 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.312909 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.313661 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.315179 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.316035 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.316193 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.319992 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.319997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.320000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.320034 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.324278 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.325092 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.325097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.325101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.325277 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.325752 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.325757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.325760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.326189 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.326375 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.326847 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.327150 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:54:51.327741 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.311364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:60966)
0619 16:54:51.311582 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.311606 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:51.311736 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:51.312168 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:51.312174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:51.312589 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:51.312748 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.312757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.313859 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.313865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.314955 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.314972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.316205 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.316210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.316420 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.317105 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:51.317124 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:51.319780 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:51.324413 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.324418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.324422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.324942 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.325392 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.325397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.325401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.325606 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.326341 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:51.326346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:51.326538 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 16:54:51.326720 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 16:54:51.327184 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 16:54:51.335744 4936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6: Bootstrap starting.
I20260619 16:54:51.326406 5017 master_runner.cc:186] Error getting master registration for 127.14.153.62:38487: OK, instance_id { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" instance_seqno: 1781888090169051 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:54:51.337059 5017 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:54:51.337688 5102 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:54:51.337828 5102 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:54:51.340380 4936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:51.341763 4936 log.cc:824] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:51.345288 4936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6: No bootstrap required, opened a new log
I20260619 16:54:51.353297 4936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } }
I20260619 16:54:51.353788 4936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c305344369044379ea5d7151c43e4d6, State: Initialized, Role: FOLLOWER
I20260619 16:54:51.354362 4936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } }
I20260619 16:54:51.355458 5105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.355899 5105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:51.356858 4936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:51.356907 4911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:54:51.357141 4911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:51.360100 4911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } }
I20260619 16:54:51.361881 4911 leader_election.cc:288] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [CANDIDATE]: Term 1 election: Requested vote from peers f1c40cb3595f45d391d76efda824dcce (127.14.153.61:40777), 5181fca92fc74b2d9afcd94ab34c4cdd (127.14.153.60:40021)
I20260619 16:54:51.362273 5059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c305344369044379ea5d7151c43e4d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd"
I20260619 16:54:51.362814 5059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:51.366888 5059 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c305344369044379ea5d7151c43e4d6 in term 1.
I20260619 16:54:51.367774 4887 leader_election.cc:302] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5181fca92fc74b2d9afcd94ab34c4cdd, 5c305344369044379ea5d7151c43e4d6; no voters:
I20260619 16:54:51.368434 5105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:51.375433 5105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [term 1 LEADER]: Becoming Leader. State: Replica: 5c305344369044379ea5d7151c43e4d6, State: Running, Role: LEADER
I20260619 16:54:51.377018 5105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } }
I20260619 16:54:51.404407 5113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c305344369044379ea5d7151c43e4d6. Latest consensus state: current_term: 1 leader_uuid: "5c305344369044379ea5d7151c43e4d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.405016 5113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [sys.catalog]: This master's current role is: LEADER
W20260619 16:54:51.408324 4890 tablet.cc:2369] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:54:51.415433 5126 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:54:51.421137 5010 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } attempt: 5
I20260619 16:54:51.421484 4869 master_runner.cc:186] Error getting master registration for 127.14.153.61:40777: OK, instance_id { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" instance_seqno: 1781888090686564 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:54:51.432112 4869 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:54:51.437633 5125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:51.440613 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.391277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.14.153.61:40777)
0619 16:54:51.391456 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.391465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.407350 (+ 15885us) server_negotiation.cc:407] Connection header received
0619 16:54:51.409511 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.409518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.409614 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.409730 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.413518 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.414718 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.417274 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.418006 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.418110 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.428736 (+ 10626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.428743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.428748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.428798 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.436264 (+ 7466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.437294 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.437300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.437303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.437488 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.438038 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.438043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.438046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.438424 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.438697 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.439604 (+ 907us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.440023 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:54:51.441761 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.391181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.14.153.62:38487)
0619 16:54:51.391391 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.391397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.408458 (+ 17061us) server_negotiation.cc:407] Connection header received
0619 16:54:51.412388 (+ 3930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.412395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.412488 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.412569 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.414774 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.416016 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.419706 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.420421 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.420532 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.430855 (+ 10323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.430865 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.430873 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.430943 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.435287 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.437493 (+ 2206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.437499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.437503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.437740 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.438653 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.438659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.438662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.439079 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.439356 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.441148 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.441394 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:54:51 dist-test-slave-qb36 krb5kdc[4856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
I20260619 16:54:51.444357 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.396050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.14.153.60:40021)
0619 16:54:51.396275 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.396283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.409056 (+ 12773us) server_negotiation.cc:407] Connection header received
0619 16:54:51.409791 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.409797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.409881 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.409952 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.417933 (+ 7981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.419018 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.422913 (+ 3895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.423918 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.424081 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.432086 (+ 8005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.432096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.432103 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.432164 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.436262 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.440398 (+ 4136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.440408 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.440415 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.440597 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.442456 (+ 1859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.442464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.442470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.442832 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.443106 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.443668 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.443916 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:54:51.450863 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.421893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:40021 (local address 127.0.0.1:59416)
0619 16:54:51.422310 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:51.422333 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:51.422441 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:51.434159 (+ 11718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:51.434168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:51.434691 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:51.434945 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.434961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.437204 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.437209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.438674 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:51.438689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.439721 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.439730 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:51.439987 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.440653 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:51.440679 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:51.443510 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:51.447777 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.447787 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.447794 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.448479 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.448917 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:51.448925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:51.448931 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:51.449133 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:51.449776 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:51.449785 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:51.449999 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 16:54:51.450167 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 16:54:51.450461 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260619 16:54:51.451810 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.422247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.14.153.60:40021)
0619 16:54:51.433656 (+ 11409us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.433667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.433692 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:54:51.433760 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.433764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.433849 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.435139 (+ 1290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.435426 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.437046 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.438867 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.439534 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.443664 (+ 4130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.443900 (+ 236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:51.443906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:51.443910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:51.443938 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:51.447640 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.448626 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.448631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.448634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.448791 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:51.449254 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:51.449258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:51.449262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:51.449628 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:51.449769 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.451180 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.451391 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11227,"thread_start_us":11675,"threads_started":1}
I20260619 16:54:51.457705 5010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce: Bootstrap starting.
I20260619 16:54:51.458068 5125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 16:54:51.458334 5038 tablet.cc:2369] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:54:51.461781 5010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:51.463151 5010 log.cc:824] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:51.472951 5059 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [term 1 FOLLOWER]: Refusing update from remote peer 5c305344369044379ea5d7151c43e4d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:51.473920 5113 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:51.475335 5010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce: No bootstrap required, opened a new log
I20260619 16:54:51.487280 5010 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } }
I20260619 16:54:51.487898 5010 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1c40cb3595f45d391d76efda824dcce, State: Initialized, Role: FOLLOWER
I20260619 16:54:51.488540 5010 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } }
I20260619 16:54:51.495658 5132 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.498047 5132 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:51.497901 5010 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:51.497978 4985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c305344369044379ea5d7151c43e4d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f1c40cb3595f45d391d76efda824dcce"
I20260619 16:54:51.498441 4984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:51.502331 4984 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [term 1 FOLLOWER]: Refusing update from remote peer 5c305344369044379ea5d7151c43e4d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:51.503504 5113 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:51.506163 4985 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c305344369044379ea5d7151c43e4d6 in term 1.
I20260619 16:54:51.506369 5091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c305344369044379ea5d7151c43e4d6. Latest consensus state: current_term: 1 leader_uuid: "5c305344369044379ea5d7151c43e4d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.506695 5091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:51.511566 5129 mvcc.cc:205] Tried to move back new op lower bound from 7298613622661464064 to 7298613622294335488. Current Snapshot: MvccSnapshot[applied={T|T < 7298613622661464064}]
I20260619 16:54:51.513561 5131 mvcc.cc:205] Tried to move back new op lower bound from 7298613622661464064 to 7298613622294335488. Current Snapshot: MvccSnapshot[applied={T|T < 7298613622661464064}]
I20260619 16:54:51.521652 5105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c305344369044379ea5d7151c43e4d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.522053 5105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:51.524960 5125 catalog_manager.cc:1347] Generated new cluster ID: ce73cb2509424afbb4026e5e46bade22
I20260619 16:54:51.525077 5125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:51.530645 5091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c305344369044379ea5d7151c43e4d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.530865 5091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:51.535457 5132 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c305344369044379ea5d7151c43e4d6. Latest consensus state: current_term: 1 leader_uuid: "5c305344369044379ea5d7151c43e4d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.535873 5132 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:51.539858 5113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c305344369044379ea5d7151c43e4d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.540148 5113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:51.550902 5132 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c305344369044379ea5d7151c43e4d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c305344369044379ea5d7151c43e4d6" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 38487 } } peers { permanent_uuid: "f1c40cb3595f45d391d76efda824dcce" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 40777 } } peers { permanent_uuid: "5181fca92fc74b2d9afcd94ab34c4cdd" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 40021 } } }
I20260619 16:54:51.551183 5132 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:51.567014 5125 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:51.569108 5125 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:51.593775 5125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c305344369044379ea5d7151c43e4d6: Generated new TSK 0
I20260619 16:54:51.595093 5125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:54:51.645843 5151 catalog_manager.cc:1259] Loaded cluster ID: ce73cb2509424afbb4026e5e46bade22
I20260619 16:54:51.655234 5151 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce: loading cluster ID for follower catalog manager: success
I20260619 16:54:51.659510 5151 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce: acquiring CA information for follower catalog manager: success
I20260619 16:54:51.662273 5151 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1c40cb3595f45d391d76efda824dcce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 16:54:51.775372 5088 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:51.765651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.14.153.62:38487)
0619 16:54:51.765896 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.765902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.773290 (+ 7388us) server_negotiation.cc:407] Connection header received
0619 16:54:51.773602 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.773608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.773697 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.773783 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.775106 (+ 1323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46752: BlockingRecv error: recv got EOF from 127.0.0.1:46752 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 16:54:51.775593 5015 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:51.772774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.14.153.61:40777)
0619 16:54:51.772984 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.772990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.773325 (+ 335us) server_negotiation.cc:407] Connection header received
0619 16:54:51.773656 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.773663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.773743 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.773825 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.775332 (+ 1507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37186: BlockingRecv error: recv got EOF from 127.0.0.1:37186 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 16:54:51.777957 5127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:51.772157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.14.153.60:40021)
0619 16:54:51.772622 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.772632 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.776123 (+ 3491us) server_negotiation.cc:407] Connection header received
0619 16:54:51.776379 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.776385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.776484 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.776553 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.777707 (+ 1154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59420: BlockingRecv error: recv got EOF from 127.0.0.1:59420 (error 108)
Metrics: {"server-negotiator.queue_time_us":269}
I20260619 16:54:51.821168 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.812845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.14.153.61:40777)
0619 16:54:51.813031 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.813038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.813145 (+ 107us) server_negotiation.cc:407] Connection header received
0619 16:54:51.813480 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.813487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.813727 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.813871 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:51.814676 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.815892 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.817938 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.818621 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.818724 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.819523 (+ 799us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:51.820289 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:51.820428 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.820690 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.820798 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 16:54:51.880055 5127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:51.863305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.14.153.60:40021)
0619 16:54:51.863535 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.863545 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.877816 (+ 14271us) server_negotiation.cc:407] Connection header received
0619 16:54:51.878369 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.878376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.878512 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.878600 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.879766 (+ 1166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59424: BlockingRecv error: recv got EOF from 127.0.0.1:59424 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:54:51.928737 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:51.910516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.14.153.61:40777)
0619 16:54:51.910702 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.910708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.921462 (+ 10754us) server_negotiation.cc:407] Connection header received
0619 16:54:51.922028 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.922035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.922128 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.922191 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:51.922993 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.924160 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.926230 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:51.926903 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:51.926998 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:51.927903 (+ 905us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:51.928053 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:51.928112 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:51.928326 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 16:54:51.928413 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 16:54:51.974376 5127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:51.964612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.14.153.60:40021)
0619 16:54:51.964847 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:51.964853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:51.972257 (+ 7404us) server_negotiation.cc:407] Connection header received
0619 16:54:51.972784 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:51.972791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:51.972897 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:51.972973 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:51.974128 (+ 1155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59434: BlockingRecv error: recv got EOF from 127.0.0.1:59434 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 16:54:52.032267 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.013573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.14.153.61:40777)
0619 16:54:52.013777 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.013783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.024670 (+ 10887us) server_negotiation.cc:407] Connection header received
0619 16:54:52.025268 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.025278 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.025390 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.025462 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.026227 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.027481 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.029591 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.030305 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.030416 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.031080 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.031233 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.031295 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.031817 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.031914 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 16:54:52.084440 5127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:52.073057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.14.153.60:40021)
0619 16:54:52.073246 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.073252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.082715 (+ 9463us) server_negotiation.cc:407] Connection header received
0619 16:54:52.082909 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.082915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.083017 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.083190 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:52.084182 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59450: BlockingRecv error: recv got EOF from 127.0.0.1:59450 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:54:52.101137 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.093787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.14.153.61:40777)
0619 16:54:52.094042 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.094049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.094620 (+ 571us) server_negotiation.cc:407] Connection header received
0619 16:54:52.094859 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.094866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.094959 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.095089 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.095619 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.096793 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.098780 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.099545 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.099654 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.100307 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.100454 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.100574 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.100728 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.100829 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260619 16:54:52.110335 5127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:52.108462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.14.153.60:40021)
0619 16:54:52.108685 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.108691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.108713 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:54:52.108998 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.109004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.109102 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.109204 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:52.110101 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59454: BlockingRecv error: recv got EOF from 127.0.0.1:59454 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 16:54:52.129470 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.121941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.14.153.61:40777)
0619 16:54:52.122139 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.122147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.122717 (+ 570us) server_negotiation.cc:407] Connection header received
0619 16:54:52.122911 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.122918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.123014 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.123156 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.123718 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.124994 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.127080 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.127884 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.128043 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.128580 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.128733 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.128825 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.129018 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.129119 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 16:54:52.149925 5127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:52.147548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.14.153.60:40021)
0619 16:54:52.147772 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.147779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.148353 (+ 574us) server_negotiation.cc:407] Connection header received
0619 16:54:52.148559 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.148564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.148672 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.148783 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:52.149702 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59462: BlockingRecv error: recv got EOF from 127.0.0.1:59462 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:54:52.176882 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.169991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.14.153.61:40777)
0619 16:54:52.170203 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.170210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.170602 (+ 392us) server_negotiation.cc:407] Connection header received
0619 16:54:52.170772 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.170779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.170877 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.170998 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.171488 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.172699 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.174563 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.175347 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.175500 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.176047 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.176204 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.176383 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.176477 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.176574 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 16:54:52.190930 5127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:52.188652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.14.153.60:40021)
0619 16:54:52.188841 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.188848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.189393 (+ 545us) server_negotiation.cc:407] Connection header received
0619 16:54:52.189603 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.189609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.189711 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.189826 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:52.190676 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59468: BlockingRecv error: recv got EOF from 127.0.0.1:59468 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:54:52.231734 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.226265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.14.153.61:40777)
0619 16:54:52.226866 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.226873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.226896 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:54:52.227168 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.227174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.227242 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.227347 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.227774 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.228506 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.229797 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.230294 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.230434 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.230847 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.230969 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.231115 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.231284 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.231367 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423}
W20260619 16:54:52.239352 5127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:52.237397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.14.153.60:40021)
0619 16:54:52.237557 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.237561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.237983 (+ 422us) server_negotiation.cc:407] Connection header received
0619 16:54:52.238147 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.238153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.238250 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.238368 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:52.239111 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59480: BlockingRecv error: recv got EOF from 127.0.0.1:59480 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:54:52.311766 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.306722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.14.153.61:40777)
0619 16:54:52.306940 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.306944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.307303 (+ 359us) server_negotiation.cc:407] Connection header received
0619 16:54:52.307485 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.307489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.307593 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.307687 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.308089 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.308832 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.310100 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.310515 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.310640 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.311068 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.311179 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.311265 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.311470 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.311541 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260619 16:54:52.319653 5127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:52.317588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.14.153.60:40021)
0619 16:54:52.317780 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.317788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.318224 (+ 436us) server_negotiation.cc:407] Connection header received
0619 16:54:52.318447 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.318451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.318526 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.318647 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:52.319486 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59494: BlockingRecv error: recv got EOF from 127.0.0.1:59494 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:54:52.339905 5102 catalog_manager.cc:1259] Loaded cluster ID: ce73cb2509424afbb4026e5e46bade22
I20260619 16:54:52.339998 5102 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd: loading cluster ID for follower catalog manager: success
I20260619 16:54:52.342471 5102 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd: acquiring CA information for follower catalog manager: success
I20260619 16:54:52.344338 5102 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5181fca92fc74b2d9afcd94ab34c4cdd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:54:52.457206 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.452422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.14.153.61:40777)
0619 16:54:52.452630 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.452635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.452976 (+ 341us) server_negotiation.cc:407] Connection header received
0619 16:54:52.453147 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.453157 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.453227 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.453318 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.453689 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.454384 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.455650 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.456038 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.456129 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.456469 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.456572 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.456636 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.456803 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.456987 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:54:52.468199 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.462636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.14.153.60:40021)
0619 16:54:52.462814 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.462818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.463213 (+ 395us) server_negotiation.cc:407] Connection header received
0619 16:54:52.463454 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.463458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.463608 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.463716 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.464112 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.464917 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.466200 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.466610 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.466714 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.467202 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.467736 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.467847 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.467919 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.467977 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:54:52.485491 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.474953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.14.153.61:40777)
0619 16:54:52.475386 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.475395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.476822 (+ 1427us) server_negotiation.cc:407] Connection header received
0619 16:54:52.477034 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.477043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.477207 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.477323 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.477992 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.479529 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.482417 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.483897 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.484072 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.484702 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.484835 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.484942 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.485105 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.485188 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260619 16:54:52.491228 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.474819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.14.153.62:38487)
0619 16:54:52.484638 (+ 9819us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.484647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.484684 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:54:52.484839 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.484847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.485137 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.485369 (+ 232us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.485941 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.486952 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.488776 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.489423 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.489558 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.490049 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.490542 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.490620 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.490798 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.490869 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9638,"thread_start_us":9504,"threads_started":1}
I20260619 16:54:52.492380 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:52.474775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.14.153.60:40021)
0619 16:54:52.475190 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:52.475199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:52.486245 (+ 11046us) server_negotiation.cc:407] Connection header received
0619 16:54:52.487115 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:52.487124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:52.487236 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:52.487334 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:52.487947 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.488678 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.490421 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:52.491083 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:52.491224 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:52.491623 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:52.491766 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:52.491852 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:52.492030 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:54:52.492091 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260619 16:54:52.496531 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4869
I20260619 16:54:52.497691 4931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:52.608987 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 4869
W20260619 16:54:52.630295 4959 connection.cc:535] server connection from 127.0.0.1:41230 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 16:54:52.630304 5036 connection.cc:535] server connection from 127.0.0.1:60966 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:54:52.630988 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 4943
I20260619 16:54:52.632139 5005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:52.738976 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 4943
I20260619 16:54:52.755879 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 5017
I20260619 16:54:52.756954 5079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:52.865579 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 5017
2026-06-19T16:54:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4530 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:54:52 dist-test-slave-qb36 krb5kdc[5266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:54:52 dist-test-slave-qb36 krb5kdc[5266](info): set up 2 sockets
Jun 19 16:54:52 dist-test-slave-qb36 krb5kdc[5266](info): commencing operation
krb5kdc: starting...
W20260619 16:54:53.983695 14948 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/test-user.keytab.
Jun 19 16:54:54 dist-test-slave-qb36 krb5kdc[5266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
Entry for principal kudu/127.14.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab.
I20260619 16:54:54.250476 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34423
--webserver_interface=127.14.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34423,127.14.153.61:44071,127.14.153.60:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:54:54.579674 5279 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:54:54.580022 5279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:54.580094 5279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:54.580173 5279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:54.587572 5279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:54.587677 5279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:54.587734 5279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:54.587795 5279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:54.587857 5279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:54.597916 5279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34423,127.14.153.61:44071,127.14.153.60:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.62.keytab
--principal=kudu/127.14.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.62:34423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:54.599221 5279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:54.600510 5279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:54.612339 5284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:54.612416 5285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:54.612391 5287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:54.613430 5279 server_base.cc:1034] running on GCE node
I20260619 16:54:54.614171 5279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:54.615671 5279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:54.616940 5279 hybrid_clock.cc:648] HybridClock initialized: now 1781888094616866 us; error 42 us; skew 500 ppm
I20260619 16:54:54.617314 5279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:54 dist-test-slave-qb36 krb5kdc[5266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:54.624504 5279 init.cc:377] Logged in from keytab as kudu/127.14.153.62@KRBTEST.COM (short username kudu)
I20260619 16:54:54.626168 5279 webserver.cc:460] Webserver started at http://127.14.153.62:40107/ using document root <none> and password file <none>
I20260619 16:54:54.626874 5279 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:54.626963 5279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:54.627300 5279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:54.630462 5279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/data/instance:
uuid: "425d6d93ad2944a1b30e94421c836823"
format_stamp: "Formatted at 2026-06-19 16:54:54 on dist-test-slave-qb36"
server_key: "e0c1c967b474e12da0e63e46d9cff1d8"
server_key_iv: "50af878022a8f461034cfb12df8cf206"
server_key_version: "encryptionkey@0"
I20260619 16:54:54.631278 5279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/wal/instance:
uuid: "425d6d93ad2944a1b30e94421c836823"
format_stamp: "Formatted at 2026-06-19 16:54:54 on dist-test-slave-qb36"
server_key: "e0c1c967b474e12da0e63e46d9cff1d8"
server_key_iv: "50af878022a8f461034cfb12df8cf206"
server_key_version: "encryptionkey@0"
I20260619 16:54:54.636508 5279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 16:54:54.640024 5294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:54.641430 5279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:54:54.641624 5279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/wal
uuid: "425d6d93ad2944a1b30e94421c836823"
format_stamp: "Formatted at 2026-06-19 16:54:54 on dist-test-slave-qb36"
server_key: "e0c1c967b474e12da0e63e46d9cff1d8"
server_key_iv: "50af878022a8f461034cfb12df8cf206"
server_key_version: "encryptionkey@0"
I20260619 16:54:54.641856 5279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:54.655653 5279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:54.659861 5279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:54.660161 5279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:54.679353 5279 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.62:34423
I20260619 16:54:54.679381 5345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.62:34423 every 8 connection(s)
I20260619 16:54:54.680856 5279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/data/info.pb
I20260619 16:54:54.684840 5346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:54.687368 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 5279
I20260619 16:54:54.687546 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-0/wal/instance
I20260619 16:54:54.688007 14948 external_mini_cluster.cc:1428] Setting key caebe34d9e5ecb078acc146cf3e5dbf2
I20260619 16:54:54.691967 5346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:54 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.153.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.61@KRBTEST.COM" created.
I20260619 16:54:54.719224 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.695300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.14.153.62:34423)
0619 16:54:54.697279 (+ 1979us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:54.697285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:54.697327 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:54:54.699717 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:54.699723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:54.699976 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:54.700150 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:54.701857 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:54.703132 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:54.705449 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:54.706179 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:54.706356 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:54.711306 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:54.711315 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:54.711330 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:54.711366 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:54.714714 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:54.715579 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:54.715587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:54.715589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:54.715776 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:54.716191 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:54.716206 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:54.716209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:54.716820 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:54.716995 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:54.717683 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 16:54:54.717996 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1773,"spinlock_wait_cycles":16768,"thread_start_us":1535,"threads_started":1}
I20260619 16:54:54.719280 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.695300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34423 (local address 127.0.0.1:39986)
0619 16:54:54.696311 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:54.696362 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:54.699279 (+ 2917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:54.700250 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:54.700260 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:54.701083 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:54.701626 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:54.701655 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:54.703300 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:54.703309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:54.705254 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:54.705274 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:54.706460 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:54.706470 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:54.706700 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:54.707542 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:54.707585 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:54.710851 (+ 3266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:54.714879 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:54.714891 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:54.714902 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:54.715401 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:54.715896 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:54.715900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:54.715902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:54.716061 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:54.716988 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:54.717000 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:54.717263 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 16:54:54.717560 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 16:54:54.718209 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":73,"thread_start_us":402,"threads_started":1}
I20260619 16:54:54.722631 5346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:54:54.723923 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.723521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34186)
0619 16:54:54.723691 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:54.723807 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 16:54:54.724318 5295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
I20260619 16:54:54.725888 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.725529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34198)
0619 16:54:54.725668 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:54.725744 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:54:54.726140 5346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:44071: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab.
Entry for principal kudu/127.14.153.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab.
I20260619 16:54:54.750144 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.61:44071
--webserver_interface=127.14.153.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab
--principal=kudu/127.14.153.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34423,127.14.153.61:44071,127.14.153.60:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:54:54.775540 5346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } attempt: 1
I20260619 16:54:54.776796 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.776352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34204)
0619 16:54:54.776547 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:54.776664 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 16:54:54.777920 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.777544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34220)
0619 16:54:54.777710 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:54.777783 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 16:54:54.778205 5346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:44071: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
I20260619 16:54:54.846483 5346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } attempt: 2
I20260619 16:54:54.847842 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.847324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34228)
0619 16:54:54.847606 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:54.847702 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 16:54:54.848958 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.848485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34244)
0619 16:54:54.848675 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:54.848785 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 16:54:54.849150 5346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:44071: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
I20260619 16:54:54.940435 5346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } attempt: 3
I20260619 16:54:54.941615 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.941155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34260)
0619 16:54:54.941379 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:54.941481 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 16:54:54.942742 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:54.942374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34270)
0619 16:54:54.942530 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:54.942598 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 16:54:54.942919 5346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:44071: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
I20260619 16:54:55.086156 5346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } attempt: 4
I20260619 16:54:55.087488 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.086877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34274)
0619 16:54:55.087251 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.087351 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260619 16:54:55.088560 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.088194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34282)
0619 16:54:55.088360 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.088439 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 16:54:55.088757 5346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.61:44071: Network error: Client connection negotiation failed: client connection to 127.14.153.61:44071: connect: Connection refused (error 111)
W20260619 16:54:55.088805 5353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:54:55.089218 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:55.089339 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:55.089488 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:54:55.097576 5353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:55.097739 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:55.097870 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:55.097956 5353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:55.098048 5353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:54:55.108371 5353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34423,127.14.153.61:44071,127.14.153.60:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.61.keytab
--principal=kudu/127.14.153.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.61:44071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.153.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:55.109831 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:55.111245 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:54:55.123437 5358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:55.123507 5361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:55.123476 5359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:55.124361 5353 server_base.cc:1034] running on GCE node
I20260619 16:54:55.125228 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:55.126775 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:55.128111 5353 hybrid_clock.cc:648] HybridClock initialized: now 1781888095127985 us; error 74 us; skew 500 ppm
I20260619 16:54:55.128510 5353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888095, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:55.132025 5353 init.cc:377] Logged in from keytab as kudu/127.14.153.61@KRBTEST.COM (short username kudu)
I20260619 16:54:55.133708 5353 webserver.cc:460] Webserver started at http://127.14.153.61:37683/ using document root <none> and password file <none>
I20260619 16:54:55.134454 5353 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:55.134588 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:55.134928 5353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:55.138167 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/data/instance:
uuid: "a28ac15fba5040c8a9d4738165d72b62"
format_stamp: "Formatted at 2026-06-19 16:54:55 on dist-test-slave-qb36"
server_key: "df23ca464f69940892f7e2ea22cb5ea8"
server_key_iv: "107a89757f636cccb1df132b2a83fce7"
server_key_version: "encryptionkey@0"
I20260619 16:54:55.139082 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/wal/instance:
uuid: "a28ac15fba5040c8a9d4738165d72b62"
format_stamp: "Formatted at 2026-06-19 16:54:55 on dist-test-slave-qb36"
server_key: "df23ca464f69940892f7e2ea22cb5ea8"
server_key_iv: "107a89757f636cccb1df132b2a83fce7"
server_key_version: "encryptionkey@0"
I20260619 16:54:55.144523 5353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 16:54:55.148535 5368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:55.150089 5353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:55.150300 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/wal
uuid: "a28ac15fba5040c8a9d4738165d72b62"
format_stamp: "Formatted at 2026-06-19 16:54:55 on dist-test-slave-qb36"
server_key: "df23ca464f69940892f7e2ea22cb5ea8"
server_key_iv: "107a89757f636cccb1df132b2a83fce7"
server_key_version: "encryptionkey@0"
I20260619 16:54:55.150544 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:55.172827 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:55.177557 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:55.177852 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:55.197300 5353 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.61:44071
I20260619 16:54:55.197319 5419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.61:44071 every 8 connection(s)
I20260619 16:54:55.198781 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/data/info.pb
I20260619 16:54:55.202577 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:55.207402 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 5353
I20260619 16:54:55.207587 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-1/wal/instance
I20260619 16:54:55.208055 14948 external_mini_cluster.cc:1428] Setting key f509e06c6543be22b8ddc8c008e17482
I20260619 16:54:55.209476 5420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888095, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:55.232944 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.212241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.14.153.62:34423)
0619 16:54:55.212469 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.212475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.213839 (+ 1364us) server_negotiation.cc:407] Connection header received
0619 16:54:55.215895 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.215902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.216005 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.216140 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.217711 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.218862 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.220509 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.221286 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.221434 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.225328 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.225337 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.225345 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.225395 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.229341 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.230317 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.230327 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.230333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.230506 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.231022 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.231050 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.231055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.231374 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.231545 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.232331 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.232568 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
WARNING: no policy specified for kudu/127.14.153.60@KRBTEST.COM; defaulting to no policy
I20260619 16:54:55.234025 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.212798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34423 (local address 127.0.0.1:39996)
0619 16:54:55.213623 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.213679 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:55.215565 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:55.216339 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:55.216346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:55.216937 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:55.217502 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.217520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.219074 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.219081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.220338 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.220358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.221508 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.221513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.221746 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.222486 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:55.222509 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:55.225064 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:55.229563 (+ 4499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.229575 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.229596 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.230151 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.230656 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.230659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.230661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.230866 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.231518 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:55.231526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:55.231927 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 16:54:55.232215 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 16:54:55.232765 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":341,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.153.60@KRBTEST.COM" created.
I20260619 16:54:55.236121 5420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888095, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
I20260619 16:54:55.253082 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.236993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34298)
0619 16:54:55.237163 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.237185 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:55.237300 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:55.238500 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:55.238511 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:55.239057 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:55.239273 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.239288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.240725 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.240733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.242220 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.242237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.243082 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.243090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.243337 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.243959 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:55.243987 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:55.246343 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:55.250709 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.250713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.250716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.251126 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.251518 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.251521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.251524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.251660 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.252403 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:55.252406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:55.252513 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 16:54:55.252614 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:54:55.252812 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:54:55.253600 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.237268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.14.153.61:44071)
0619 16:54:55.238010 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.238016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.238040 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:54:55.238085 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.238091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.238285 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.238402 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.239486 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.240563 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.242371 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.242905 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.243001 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.246555 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.246571 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.246591 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.246634 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.250567 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.251249 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.251261 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.251267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.251413 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.251774 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.251781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.251787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.252282 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.252501 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.252805 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.253100 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":446,"threads_started":1}
I20260619 16:54:55.256439 5420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:54:55.257658 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.257069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36674)
0619 16:54:55.257295 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.257438 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 16:54:55.257928 5369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
I20260619 16:54:55.259610 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.259184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36680)
0619 16:54:55.259354 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.259464 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 16:54:55.259840 5420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:42685: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.153.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab.
Entry for principal kudu/127.14.153.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab.
I20260619 16:54:55.270514 14948 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
/tmp/dist-test-taskGGaEbW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.60:42685
--webserver_interface=127.14.153.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab
--principal=kudu/127.14.153.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.153.20:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.153.62:34423,127.14.153.61:44071,127.14.153.60:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:54:55.309203 5420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } attempt: 1
I20260619 16:54:55.310461 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.309915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36694)
0619 16:54:55.310144 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.310274 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:54:55.311556 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.311171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36704)
0619 16:54:55.311338 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.311427 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 16:54:55.311805 5420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:42685: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
I20260619 16:54:55.380038 5420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } attempt: 2
I20260619 16:54:55.381404 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.380753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36718)
0619 16:54:55.381021 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.381168 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 16:54:55.382685 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.382172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36720)
0619 16:54:55.382382 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.382493 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 16:54:55.382997 5420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:42685: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
I20260619 16:54:55.388010 5346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } attempt: 5
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
I20260619 16:54:55.401674 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.388842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.14.153.61:44071)
0619 16:54:55.389007 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.389015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.389083 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:54:55.389309 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.389314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.389394 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.389507 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.390299 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.391007 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.392188 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.392597 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.392735 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.396100 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.396105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.396108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.396141 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.398960 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.399704 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.399711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.399714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.399846 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.400233 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.400239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.400243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.400536 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.400706 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.401088 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.401318 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:54:55.401674 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.388759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34302)
0619 16:54:55.388972 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.388993 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:55.389149 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:55.389536 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:55.389541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:55.389954 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:55.390135 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.390144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.391177 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.391182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.392029 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.392049 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.392764 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.392770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.392984 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.393628 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:55.393645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:55.395900 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:55.399164 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.399173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.399178 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.399573 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.399964 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.399969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.399973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.400119 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.400664 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:55.400669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:55.400835 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 16:54:55.400981 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:54:55.401318 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:54:55.402750 5346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:54:55.403801 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.403407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36730)
0619 16:54:55.403585 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.403660 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 16:54:55.404819 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.404456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36734)
0619 16:54:55.404641 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.404720 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 16:54:55.405027 5346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:42685: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
I20260619 16:54:55.456234 5346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } attempt: 1
I20260619 16:54:55.457381 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.456913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36744)
0619 16:54:55.457112 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.457211 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 16:54:55.458371 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.458056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36750)
0619 16:54:55.458193 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.458271 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:54:55.458575 5346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:42685: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
I20260619 16:54:55.474300 5420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } attempt: 3
I20260619 16:54:55.475737 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.475106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36754)
0619 16:54:55.475362 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.475499 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:54:55.476935 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.476404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36756)
0619 16:54:55.476669 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.476772 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260619 16:54:55.477175 5420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:42685: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
I20260619 16:54:55.526796 5346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } attempt: 2
I20260619 16:54:55.528014 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.527491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36772)
0619 16:54:55.527764 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.527853 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 16:54:55.529074 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.528759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36786)
0619 16:54:55.528907 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.528982 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 16:54:55.529259 5346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:42685: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
W20260619 16:54:55.613735 5427 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:54:55.614140 5427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:54:55.614267 5427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:54:55.614403 5427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:54:55.620404 5420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } attempt: 4
I20260619 16:54:55.621644 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.621087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36802)
0619 16:54:55.621341 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.621447 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:54:55.622653 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.622305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36810)
0619 16:54:55.622448 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.622526 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 16:54:55.622675 5427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:54:55.622761 5427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:54:55.622848 5427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:54:55.622947 5427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:54:55.623059 5427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:54:55.622972 5420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:42685: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
I20260619 16:54:55.633131 5427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.153.20:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.153.62:34423,127.14.153.61:44071,127.14.153.60:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/krb5kdc/kudu_127.14.153.60.keytab
--principal=kudu/127.14.153.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.153.60:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.153.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 15:53:41 UTC on ccycloud.quasar-tzgprr.root.comops.site
build id 4456155
ASAN enabled
I20260619 16:54:55.634485 5427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:54:55.635450 5346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } attempt: 3
I20260619 16:54:55.636128 5427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:54:55.636624 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.636067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36826)
0619 16:54:55.636310 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.636423 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 16:54:55.637758 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.637359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36830)
0619 16:54:55.637519 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.637622 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 16:54:55.637984 5346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.153.60:42685: Network error: Client connection negotiation failed: client connection to 127.14.153.60:42685: connect: Connection refused (error 111)
W20260619 16:54:55.648645 5433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:55.648728 5435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:54:55.648705 5432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:54:55.649919 5427 server_base.cc:1034] running on GCE node
I20260619 16:54:55.650658 5427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:54:55.652101 5427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:54:55.653323 5427 hybrid_clock.cc:648] HybridClock initialized: now 1781888095653237 us; error 57 us; skew 500 ppm
I20260619 16:54:55.653669 5427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888095, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:54:55.657210 5427 init.cc:377] Logged in from keytab as kudu/127.14.153.60@KRBTEST.COM (short username kudu)
I20260619 16:54:55.658607 5427 webserver.cc:460] Webserver started at http://127.14.153.60:36121/ using document root <none> and password file <none>
I20260619 16:54:55.659366 5427 fs_manager.cc:359] Metadata directory not provided
I20260619 16:54:55.659490 5427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:54:55.659834 5427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:54:55.662881 5427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/data/instance:
uuid: "af3e7efda30b4a7bae2be7a225493163"
format_stamp: "Formatted at 2026-06-19 16:54:55 on dist-test-slave-qb36"
server_key: "a31a8e3100c81d4f1a23437e221520e2"
server_key_iv: "1a657a7749bd31c3df586ebde2506c36"
server_key_version: "encryptionkey@0"
I20260619 16:54:55.663756 5427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/wal/instance:
uuid: "af3e7efda30b4a7bae2be7a225493163"
format_stamp: "Formatted at 2026-06-19 16:54:55 on dist-test-slave-qb36"
server_key: "a31a8e3100c81d4f1a23437e221520e2"
server_key_iv: "1a657a7749bd31c3df586ebde2506c36"
server_key_version: "encryptionkey@0"
I20260619 16:54:55.668979 5427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:54:55.672456 5442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:54:55.673807 5427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:54:55.673996 5427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/data,/tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/wal
uuid: "af3e7efda30b4a7bae2be7a225493163"
format_stamp: "Formatted at 2026-06-19 16:54:55 on dist-test-slave-qb36"
server_key: "a31a8e3100c81d4f1a23437e221520e2"
server_key_iv: "1a657a7749bd31c3df586ebde2506c36"
server_key_version: "encryptionkey@0"
I20260619 16:54:55.674254 5427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:54:55.703323 5427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:54:55.707865 5427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:54:55.708185 5427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:54:55.726594 5427 rpc_server.cc:307] RPC server started. Bound to: 127.14.153.60:42685
I20260619 16:54:55.726641 5493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.153.60:42685 every 8 connection(s)
I20260619 16:54:55.728087 5427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/data/info.pb
I20260619 16:54:55.732025 5494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:54:55.737056 14948 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu as pid 5427
I20260619 16:54:55.737264 14948 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGGaEbW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887734944185-14948-0/minicluster-data/master-2/wal/instance
I20260619 16:54:55.737672 14948 external_mini_cluster.cc:1428] Setting key 8930a41b2ae2376530096954083f0ac8
I20260619 16:54:55.739177 5494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888095, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
I20260619 16:54:55.761708 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.741862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.14.153.62:34423)
0619 16:54:55.743349 (+ 1487us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.743357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.743380 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:54:55.743444 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.743449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.743524 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.743641 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.745391 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.746714 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.748976 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.749621 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.749757 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.754311 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.754317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.754321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.754359 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.757995 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.758919 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.758923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.758925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.759134 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.759583 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.759587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.759589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.759941 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.760149 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.761072 (+ 923us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.761318 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1354,"thread_start_us":1248,"threads_started":1}
I20260619 16:54:55.762693 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.739496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.14.153.60:42685)
0619 16:54:55.740321 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.740333 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.740374 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:54:55.742676 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.742706 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.743208 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.743465 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.745391 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.747234 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.749021 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.749918 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.750064 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.754723 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.754736 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.754757 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.754804 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.758292 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.759192 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.759204 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.759210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.759389 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.759814 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.759819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.759822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.760499 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.760788 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.761215 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.761857 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":615,"thread_start_us":397,"threads_started":1}
I20260619 16:54:55.762712 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.741814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.62:34423 (local address 127.0.0.1:40010)
0619 16:54:55.742762 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.742819 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:55.743165 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:55.743731 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:55.743740 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:55.744637 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:55.745228 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.745257 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.746908 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.746912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.748797 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.748812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.749866 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.749871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.750096 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.750914 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:55.750946 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:55.754022 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:55.758164 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.758174 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.758191 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.758782 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.759258 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.759264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.759267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.759466 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.760113 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:55.760120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:55.760531 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 16:54:55.760940 (+ 409us) client_negotiation.cc:240] Negotiation successful
0619 16:54:55.761491 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":64,"spinlock_wait_cycles":15744,"thread_start_us":380,"threads_started":1}
I20260619 16:54:55.765297 5494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888095, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.62@KRBTEST.COM
I20260619 16:54:55.779748 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.766059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.61:44071 (local address 127.0.0.1:34308)
0619 16:54:55.766323 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.766341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:55.766456 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:55.766868 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:55.766873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:55.767388 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:55.767611 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.767625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.768803 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.768806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.770064 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.770078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.770781 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.770786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.770913 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.771575 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:55.771592 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:55.773578 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:55.776902 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.776910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.776918 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.777474 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.777924 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.777932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.777938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.778121 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.778712 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:55.778717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:55.778887 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 16:54:55.779066 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:54:55.779378 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 16:54:55.779779 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.766196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.14.153.61:44071)
0619 16:54:55.766362 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.766367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.766382 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:54:55.766604 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.766609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.766694 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.766822 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.767773 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.768653 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.770204 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.770645 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.770773 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.773795 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.773802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.773810 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.773856 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.776757 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.777650 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.777654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.777657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.777802 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.778263 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.778267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.778269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.778562 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.778816 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.779153 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.779373 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:54:55.781596 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.766078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.14.153.62:34423)
0619 16:54:55.766263 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.766268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.766479 (+ 211us) server_negotiation.cc:407] Connection header received
0619 16:54:55.766621 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.766625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.766704 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.766811 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.768241 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.769105 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.770622 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.771231 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.771327 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.775403 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.775408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.775412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.775444 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.778509 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.779278 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.779284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.779288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.779439 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.779919 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.779926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.779929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.780354 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.780667 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.780905 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.781177 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:54:55.785162 5494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.153.61@KRBTEST.COM
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888095, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.60@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
I20260619 16:54:55.800400 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.783694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.14.153.61:44071)
0619 16:54:55.783865 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.783870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.783888 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:54:55.783942 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.783950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.784032 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.784132 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.786003 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.787259 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.788935 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.789553 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.789644 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.794967 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.794972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.794975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.795007 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.797590 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.798370 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.798373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.798376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.798493 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.798890 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.798896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.798899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.799226 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.799508 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.799793 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.800019 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:54:55.801019 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.785995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36846)
0619 16:54:55.786176 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.786203 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:55.786304 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:55.787560 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:55.787565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:55.788060 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:55.788254 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.788265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.789343 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.789347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.790517 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.790531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.791285 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.791292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.791547 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.792079 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:55.792106 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:55.795124 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:55.798565 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.798572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.798576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.798920 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.799323 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.799330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.799334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.799462 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.799971 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:55.799975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:55.800115 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 16:54:55.800292 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 16:54:55.800609 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:54:55.801131 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.786982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.14.153.60:42685)
0619 16:54:55.787162 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.787168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.787186 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:54:55.787237 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.787242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.787317 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.787413 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.788400 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.789211 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.790661 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.791126 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.791227 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.795311 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.795316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.795319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.795345 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.798436 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.799074 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.799080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.799083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.799212 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.799571 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.799576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.799579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.799832 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.799980 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.800405 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.800787 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":67}
I20260619 16:54:55.805423 14948 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:54:55.806572 5494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163: Bootstrap starting.
I20260619 16:54:55.809774 5494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:55.807807 5321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "425d6d93ad2944a1b30e94421c836823"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40016
I20260619 16:54:55.810844 5494 log.cc:824] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:55.813560 5494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163: No bootstrap required, opened a new log
I20260619 16:54:55.815203 5346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } attempt: 4
I20260619 16:54:55.823405 5494 raft_consensus.cc:348] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } }
I20260619 16:54:55.823789 5494 raft_consensus.cc:729] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3e7efda30b4a7bae2be7a225493163, State: Initialized, Role: FOLLOWER
I20260619 16:54:55.824285 5494 consensus_queue.cc:260] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } }
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.62@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
I20260619 16:54:55.826875 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:55.827286 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:55.828267 5494 sys_catalog.cc:564] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:55.833067 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.815902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36854)
0619 16:54:55.816121 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.816144 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:55.816281 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:55.816667 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:55.816674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:55.817242 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:55.817469 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.817482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.819008 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.819016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.820621 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.820639 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.822212 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.822253 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.822496 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.823212 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:55.823236 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:55.826081 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:55.829931 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.829936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.829941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.830457 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.830912 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.830919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.830923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.831133 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.831899 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:55.831902 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:55.832088 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 16:54:55.832257 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 16:54:55.832610 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 16:54:55.833271 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.815967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.14.153.60:42685)
0619 16:54:55.816205 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.816211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.816240 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:54:55.816411 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.816416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.816500 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.816617 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.817629 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.818835 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.820822 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.821609 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.821745 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.826304 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.826311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.826315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.826358 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.829786 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.830612 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.830625 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.830629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.830787 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.831271 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.831277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.831280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.831770 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.831935 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.832432 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.832741 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260619 16:54:55.839600 5512 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:54:55.839812 5512 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:54:55.839908 5346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823: Bootstrap starting.
I20260619 16:54:55.842932 5346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:55.844058 5346 log.cc:824] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:55.847489 5346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823: No bootstrap required, opened a new log
I20260619 16:54:55.836835 5427 master_runner.cc:186] Error getting master registration for 127.14.153.62:34423: OK, instance_id { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" instance_seqno: 1781888094673659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:54:55.848711 5427 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:54:55.855358 5346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } }
I20260619 16:54:55.855841 5346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425d6d93ad2944a1b30e94421c836823, State: Initialized, Role: FOLLOWER
I20260619 16:54:55.856355 5346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } }
I20260619 16:54:55.857614 5515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:55.858253 5515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:55.858937 5346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:55.859014 5321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:54:55.859232 5321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:55.863611 5321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } }
I20260619 16:54:55.865449 5321 leader_election.cc:288] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [CANDIDATE]: Term 1 election: Requested vote from peers a28ac15fba5040c8a9d4738165d72b62 (127.14.153.61:44071), af3e7efda30b4a7bae2be7a225493163 (127.14.153.60:42685)
I20260619 16:54:55.866672 5469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "425d6d93ad2944a1b30e94421c836823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af3e7efda30b4a7bae2be7a225493163"
I20260619 16:54:55.867249 5469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:55.871752 5469 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 425d6d93ad2944a1b30e94421c836823 in term 1.
I20260619 16:54:55.872596 5295 leader_election.cc:302] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 425d6d93ad2944a1b30e94421c836823, af3e7efda30b4a7bae2be7a225493163; no voters:
I20260619 16:54:55.873383 5515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:54:55.874179 5515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [term 1 LEADER]: Becoming Leader. State: Replica: 425d6d93ad2944a1b30e94421c836823, State: Running, Role: LEADER
I20260619 16:54:55.875063 5515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } }
I20260619 16:54:55.879241 5528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425d6d93ad2944a1b30e94421c836823. Latest consensus state: current_term: 1 leader_uuid: "425d6d93ad2944a1b30e94421c836823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:55.880369 5528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [sys.catalog]: This master's current role is: LEADER
W20260619 16:54:55.888482 5530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:54:55.888626 5530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:54:55.888819 5531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:54:55.891857 5531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:54:55.888317 5279 master_runner.cc:186] Error getting master registration for 127.14.153.61:44071: OK, instance_id { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" instance_seqno: 1781888095191830 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:54:55.901535 5279 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:54:55.909781 5469 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [term 1 FOLLOWER]: Refusing update from remote peer 425d6d93ad2944a1b30e94421c836823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:55.911284 5528 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:54:55.913074 5300 tablet.cc:2369] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:54:55.922227 5420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } attempt: 5
Jun 19 16:54:55 dist-test-slave-qb36 krb5kdc[5266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888095, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.153.61@KRBTEST.COM for kudu/127.14.153.60@KRBTEST.COM
I20260619 16:54:55.945477 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.908894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.14.153.62:34423)
0619 16:54:55.909096 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.909102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.916390 (+ 7288us) server_negotiation.cc:407] Connection header received
0619 16:54:55.927166 (+ 10776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.927173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.927263 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.927347 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.929609 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.930758 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.932771 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.933501 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.933608 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.937927 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.937937 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.937943 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.938033 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.941967 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.942869 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.942875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.942879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.943071 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.943568 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.943573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.943576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.943972 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.944217 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.944776 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.945023 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:54:55.946698 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.922894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.153.60:42685 (local address 127.0.0.1:36862)
0619 16:54:55.923291 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 16:54:55.923311 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:54:55.923431 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:54:55.924339 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:54:55.924344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:54:55.924853 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:54:55.925065 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.925078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.926690 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.926695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.928067 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:54:55.928106 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.929634 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.929640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:54:55.929891 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.930662 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:54:55.930684 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:54:55.934220 (+ 3536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:54:55.940520 (+ 6300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.940527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.940532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.941533 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.942457 (+ 924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:54:55.942462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:54:55.942465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:54:55.942774 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:54:55.944125 (+ 1351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:54:55.944133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:54:55.944454 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 16:54:55.944687 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 16:54:55.945359 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260619 16:54:55.948467 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.923630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.14.153.60:42685)
0619 16:54:55.923805 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.923813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.923830 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:54:55.923887 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.923893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.924006 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.924105 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.925256 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.926375 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.928455 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.929184 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.929353 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.934530 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.934538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.934544 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.934591 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.940258 (+ 5667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.941866 (+ 1608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.941876 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.941880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.942238 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.943080 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.943085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.943088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.943848 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.946053 (+ 2205us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.946784 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.947747 (+ 963us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:54:55.954782 5420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62: Bootstrap starting.
W20260619 16:54:55.960175 5448 tablet.cc:2369] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:54:55.965361 5420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62: Neither blocks nor log segments found. Creating new log.
I20260619 16:54:55.971275 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.913183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.14.153.61:44071)
0619 16:54:55.913375 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.913383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.923424 (+ 10041us) server_negotiation.cc:407] Connection header received
0619 16:54:55.937008 (+ 13584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.937014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.937097 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.937170 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.943920 (+ 6750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.945096 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.950333 (+ 5237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.951098 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.951200 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.961083 (+ 9883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.961091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.961096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.961139 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.964974 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.966733 (+ 1759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.966741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.966744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.966904 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.969308 (+ 2404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.969314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.969318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.969739 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.970009 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.970574 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.970856 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:54:55.972538 5420 log.cc:824] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62: Log is configured to *not* fsync() on all Append() calls
I20260619 16:54:55.976799 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:55.924053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.14.153.60:42685)
0619 16:54:55.941467 (+ 17414us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:55.941474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:55.941510 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:54:55.941571 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:55.941577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:55.941681 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:55.941786 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:55.946433 (+ 4647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.947779 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.952207 (+ 4428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:55.952930 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:55.953059 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:55.958810 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:54:55.958818 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:54:55.958823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:54:55.958866 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:54:55.968237 (+ 9371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.973411 (+ 5174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.973418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.973421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.973663 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:54:55.974145 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:54:55.974150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:54:55.974158 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:54:55.974544 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:54:55.974836 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:55.976069 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0619 16:54:55.976329 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17245,"thread_start_us":19681,"threads_started":1}
I20260619 16:54:55.986166 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425d6d93ad2944a1b30e94421c836823. Latest consensus state: current_term: 1 leader_uuid: "425d6d93ad2944a1b30e94421c836823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:55.986475 5420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62: No bootstrap required, opened a new log
I20260619 16:54:55.986502 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:55.989538 5535 mvcc.cc:205] Tried to move back new op lower bound from 7298613640831049728 to 7298613640709533696. Current Snapshot: MvccSnapshot[applied={T|T < 7298613640831049728}]
I20260619 16:54:56.007138 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "425d6d93ad2944a1b30e94421c836823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:56.007462 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:56.008061 5420 raft_consensus.cc:348] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } }
I20260619 16:54:56.008507 5420 raft_consensus.cc:729] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28ac15fba5040c8a9d4738165d72b62, State: Initialized, Role: FOLLOWER
I20260619 16:54:56.009014 5420 consensus_queue.cc:260] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } }
I20260619 16:54:56.011442 5528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "425d6d93ad2944a1b30e94421c836823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:56.011731 5528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:56.014724 5531 catalog_manager.cc:1347] Generated new cluster ID: 43b92c36aa1f407895679c0a2cf62ba3
I20260619 16:54:56.014840 5531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:54:56.016574 5545 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:56.016986 5545 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:56.018852 5420 sys_catalog.cc:564] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:54:56.022241 5395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "425d6d93ad2944a1b30e94421c836823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a28ac15fba5040c8a9d4738165d72b62"
I20260619 16:54:56.022747 5395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:54:56.032821 5395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 425d6d93ad2944a1b30e94421c836823 in term 1.
I20260619 16:54:56.034009 5394 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [term 1 FOLLOWER]: Refusing update from remote peer 425d6d93ad2944a1b30e94421c836823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:54:56.040763 5546 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:54:56.088387 5549 mvcc.cc:205] Tried to move back new op lower bound from 7298613640831049728 to 7298613640709533696. Current Snapshot: MvccSnapshot[applied={T|T < 7298613640831049728}]
I20260619 16:54:56.099287 5545 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425d6d93ad2944a1b30e94421c836823. Latest consensus state: current_term: 1 leader_uuid: "425d6d93ad2944a1b30e94421c836823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:56.099916 5545 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:56.099287 5551 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "425d6d93ad2944a1b30e94421c836823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:56.100570 5551 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:54:56.169657 5515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "425d6d93ad2944a1b30e94421c836823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "425d6d93ad2944a1b30e94421c836823" member_type: VOTER last_known_addr { host: "127.14.153.62" port: 34423 } } peers { permanent_uuid: "a28ac15fba5040c8a9d4738165d72b62" member_type: VOTER last_known_addr { host: "127.14.153.61" port: 44071 } } peers { permanent_uuid: "af3e7efda30b4a7bae2be7a225493163" member_type: VOTER last_known_addr { host: "127.14.153.60" port: 42685 } } }
I20260619 16:54:56.169984 5515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823 [sys.catalog]: This master's current role is: LEADER
I20260619 16:54:56.173849 5531 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:54:56.176975 5531 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:54:56.190029 5561 catalog_manager.cc:1259] Loaded cluster ID: 43b92c36aa1f407895679c0a2cf62ba3
I20260619 16:54:56.190158 5561 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62: loading cluster ID for follower catalog manager: success
I20260619 16:54:56.193727 5561 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62: acquiring CA information for follower catalog manager: success
W20260619 16:54:56.195012 5561 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260619 16:54:56.208873 5531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 425d6d93ad2944a1b30e94421c836823: Generated new TSK 0
I20260619 16:54:56.210160 5531 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:54:56.361344 5499 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.337712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.14.153.62:34423)
0619 16:54:56.337904 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.337911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.357667 (+ 19756us) server_negotiation.cc:407] Connection header received
0619 16:54:56.358627 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.358635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.358736 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.358825 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.361063 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40028: BlockingRecv error: recv got EOF from 127.0.0.1:40028 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 16:54:56.365850 5539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.356001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.14.153.60:42685)
0619 16:54:56.356278 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.356285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.363126 (+ 6841us) server_negotiation.cc:407] Connection header received
0619 16:54:56.364401 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.364407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.364494 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.364577 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.365584 (+ 1007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36876: BlockingRecv error: recv got EOF from 127.0.0.1:36876 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 16:54:56.367080 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.339455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.14.153.61:44071)
0619 16:54:56.339680 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.339687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.362556 (+ 22869us) server_negotiation.cc:407] Connection header received
0619 16:54:56.363624 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.363631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.363729 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.363811 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.366830 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34348: BlockingRecv error: recv got EOF from 127.0.0.1:34348 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 16:54:56.413463 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.398783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.14.153.61:44071)
0619 16:54:56.399068 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.399076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.411072 (+ 11996us) server_negotiation.cc:407] Connection header received
0619 16:54:56.411727 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.411734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.411889 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.411979 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.413126 (+ 1147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34350: BlockingRecv error: recv got EOF from 127.0.0.1:34350 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 16:54:56.457934 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.444426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.14.153.61:44071)
0619 16:54:56.444655 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.444662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.456118 (+ 11456us) server_negotiation.cc:407] Connection header received
0619 16:54:56.456443 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.456449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.456558 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.456651 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.457700 (+ 1049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34352: BlockingRecv error: recv got EOF from 127.0.0.1:34352 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 16:54:56.508278 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.495461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.14.153.61:44071)
0619 16:54:56.495683 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.495690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.506231 (+ 10541us) server_negotiation.cc:407] Connection header received
0619 16:54:56.506764 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.506771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.506883 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.506963 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.507995 (+ 1032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34358: BlockingRecv error: recv got EOF from 127.0.0.1:34358 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 16:54:56.535058 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.532908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.14.153.61:44071)
0619 16:54:56.533093 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.533097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.533556 (+ 459us) server_negotiation.cc:407] Connection header received
0619 16:54:56.533798 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.533804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.533899 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.534021 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.534768 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34370: BlockingRecv error: recv got EOF from 127.0.0.1:34370 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 16:54:56.548796 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.546849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.14.153.61:44071)
0619 16:54:56.547101 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.547108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.547439 (+ 331us) server_negotiation.cc:407] Connection header received
0619 16:54:56.547620 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.547626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.547729 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.547855 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.548537 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34384: BlockingRecv error: recv got EOF from 127.0.0.1:34384 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 16:54:56.570205 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.568240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.14.153.61:44071)
0619 16:54:56.568447 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.568456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.568750 (+ 294us) server_negotiation.cc:407] Connection header received
0619 16:54:56.568935 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.568941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.569062 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.569195 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.569939 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34394: BlockingRecv error: recv got EOF from 127.0.0.1:34394 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 16:54:56.607645 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.605608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.14.153.61:44071)
0619 16:54:56.605815 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.605822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.606155 (+ 333us) server_negotiation.cc:407] Connection header received
0619 16:54:56.606319 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.606330 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.606434 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.606631 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.607359 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34406: BlockingRecv error: recv got EOF from 127.0.0.1:34406 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 16:54:56.676975 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.674942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.14.153.61:44071)
0619 16:54:56.675722 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.675728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.675748 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:54:56.675821 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.675826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.675933 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.676043 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.676756 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34410: BlockingRecv error: recv got EOF from 127.0.0.1:34410 (error 108)
Metrics: {"server-negotiator.queue_time_us":652}
W20260619 16:54:56.810603 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:56.808698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.14.153.61:44071)
0619 16:54:56.808911 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:56.808916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:56.809371 (+ 455us) server_negotiation.cc:407] Connection header received
0619 16:54:56.809512 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:56.809516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:56.809586 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:56.809680 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:56.810404 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34424: BlockingRecv error: recv got EOF from 127.0.0.1:34424 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:54:56.841904 5512 catalog_manager.cc:1259] Loaded cluster ID: 43b92c36aa1f407895679c0a2cf62ba3
I20260619 16:54:56.842046 5512 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163: loading cluster ID for follower catalog manager: success
I20260619 16:54:56.844647 5512 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163: acquiring CA information for follower catalog manager: success
I20260619 16:54:56.846781 5512 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af3e7efda30b4a7bae2be7a225493163: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 16:54:57.078991 5424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:54:57.077022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.14.153.61:44071)
0619 16:54:57.077249 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:57.077256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:57.077487 (+ 231us) server_negotiation.cc:407] Connection header received
0619 16:54:57.077698 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:57.077703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:57.077777 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:57.077886 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:54:57.078803 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34440: BlockingRecv error: recv got EOF from 127.0.0.1:34440 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 16:54:57.197814 5561 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a28ac15fba5040c8a9d4738165d72b62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:54:57.602455 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:57.596825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.14.153.61:44071)
0619 16:54:57.597307 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:57.597312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:57.597437 (+ 125us) server_negotiation.cc:407] Connection header received
0619 16:54:57.597605 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:57.597609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:57.597768 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:57.597887 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:57.598325 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.599056 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.600401 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.600821 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.600916 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:57.601385 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:57.601929 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:57.602038 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:57.602135 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 16:54:57.602218 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
I20260619 16:54:57.614611 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:57.608353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.14.153.60:42685)
0619 16:54:57.608846 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:57.608853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:57.608949 (+ 96us) server_negotiation.cc:407] Connection header received
0619 16:54:57.609218 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:57.609222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:57.609381 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:57.609565 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:57.609953 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.610668 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.612086 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.612493 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.612645 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:57.613407 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:57.613925 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:57.614045 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:57.614179 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 16:54:57.614260 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
I20260619 16:54:57.628427 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:57.620791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.14.153.61:44071)
0619 16:54:57.621014 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:57.621024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:57.622267 (+ 1243us) server_negotiation.cc:407] Connection header received
0619 16:54:57.622558 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:57.622563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:57.622650 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:57.622731 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:57.623615 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.624612 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.626475 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.626876 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.627011 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:57.627618 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:57.627781 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:57.627894 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:57.628015 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 16:54:57.628099 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:54:57.629271 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:57.620733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.14.153.62:34423)
0619 16:54:57.621770 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:57.621778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:57.621805 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:54:57.622082 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:57.622089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:57.622348 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:57.622532 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:57.623422 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.624574 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.626349 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.626993 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.627138 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:57.627814 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:57.628287 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:57.628401 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:57.628862 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 16:54:57.628953 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":798,"threads_started":1}
I20260619 16:54:57.629912 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:54:57.622449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.14.153.60:42685)
0619 16:54:57.622665 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:54:57.622674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:54:57.623390 (+ 716us) server_negotiation.cc:407] Connection header received
0619 16:54:57.623724 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:54:57.623730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:54:57.623819 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:54:57.623910 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:54:57.624487 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.625567 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.627804 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:54:57.628419 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:54:57.628552 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:54:57.629011 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:54:57.629141 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:54:57.629229 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:54:57.629442 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 16:54:57.629527 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:54:57.633924 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 5279
I20260619 16:54:57.635156 5341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:57.747161 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 5279
I20260619 16:54:57.765028 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 5353
I20260619 16:54:57.766187 5415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:57.878289 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 5353
I20260619 16:54:57.899006 14948 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu pid 5427
I20260619 16:54:57.900221 5489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 16:54:58.007366 14948 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGGaEbW/build/asan/bin/kudu with pid 5427
2026-06-19T16:54:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5141 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9671 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (363030 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST